Analysis and Design of Algorithms

Instructor: Prof. Dr. Abeer Mahmoud

Computer science department,

Faculty of computer and information sciences,
Ain shams university.

1
 Lecture 7

Dynamic
Programming

2
 follow definition /
Brute Force & try all possibilities
Exhaustive
Search
break problem
Divide & into distinct
Iterative Conquer sub-problems
Improvement

repeatedly improve Algorithm

current solution
Design Transformation
convert
problem to
Techniques another one

Randomization

use Dynamic break problem

random Programming into overlapping
numbers sub-problems
repeatedly do
Greedy what is best now
3
 Dynamic Programming
 A strategy for designing algorithms
 The word “programming” is historical and predates computer
programming
 Dynamic programming is an algorithm design method that can be
used when the solution to a problem can be viewed as the result of
a sequence of decisions.

 Typically applied to optimization problems

• Use when problem breaks down into recurring small

Overlapping sub-problems  So, What is the difference
between Divide & Concur and Dynamic Programming?
4
 Dynamic Programming
 Divide-and-conquer is a top-down method.

 Dynamic programming on the other hand is a bottom-up

technique. We usually start with the simplest, and hence the
simplest, sub instances. By combining their solutions, we obtain
the answers to sub instances of increasing size, until finally we
arrive at the solution of the original instance.

5
 Dynamic Programming

 Divide and conquer

 Break problem into independent subproblems
 Recursively solve subproblems (subproblems are smaller
instances of main problem)
 Combine solutions

 Dynamic Programming: Appropriate when you have

recursive subproblems that are not independent
(subproblems that share subproblems)

6
The steps in the development of a dynamic
programming algorithm are as follows:

1. Establish a recursive property that gives the solution to

an instance of the problem.

2. Solve an instance of the problem in a bottom-up fashion

by solving smaller instances first.

7
Example: Fibonacci Numbers

Fibonacci numbers: 0, 1, 1, 2, 3, 5, 8, 13, 21, …

F(0)=0, F(1)=1
F(n)=F(n-1)+F(n-2) for n>1

ALGORITHM F(n) ALGORITHM F(n)

if n≤1 return n F[0] ← 0; F[1] ← 1;
else for i ← 2 to n do
return F(n-1)+F(n-2) F[i] ← F[i-1] + F[i-2];
return F[n]

8
Overlapping subproblems

Fib(5)
Fib(4) Fib(3)

Fib(3) Fib(2) Fib(2) Fib(1)

Fib(2) Fib(1) Fib(1) Fib(0) Fib(1) Fib(0)

Fib(1) Fib(0)
If all leaves had the same depth, then there
9
would be about 2n recursive calls.
Save Work
 Wasteful approach - repeat work unnecessarily
 Ex : Fib(2) is computed three times
 Instead, compute Fib(2) once, store result in a table,
and access it when needed

10
More Efficient Recursive Alg
 F[0] := 0; F[1] := 1; F[n] := Fib(n);
 Fib(n):
 if n = 0 or 1 then return F[n]
 if F[n-1] = NIL then F[n-1] := Fib(n-1)
 if F[n-2] = NIL then F[n-2] := Fib(n-2)
 return (F[n-1] + F[n-2])

 computes each F[i] only once

11
Example of Memoized Fib

F fills in F[4] with 3,

0 0 Fib(5) returns 3+2 = 5
1 1
fills in F[3] with 2,
2 1
NIL Fib(4) returns 2+1 = 3
3 2
NIL
4 3
NIL
Fib(3)
fills in F[2] with 1,
5 5
NIL returns 1+1 = 2

Fib(2) returns 0+1 = 1

12
Get Rid of the Recursion
 Recursion adds overhead
 extra time for function calls
 extra space to store information on the runtime stack about
each currently active function call
 Avoid the recursion overhead by filling in the table
entries bottom up, instead of top down.

13
Subproblem Dependencies
 Figure out which subproblems rely on which other
subproblems
 Example:

F0 F1 F2 F3 … Fn-2 Fn-1 Fn

14
Order for Computing Subproblems
 Then figure out an order for computing the
subproblems that respects the dependencies:
 when you are solving a subproblem, you have already solved
all the subproblems on which it depends
 Example: Just solve them in the order
F0 , F1 , F2 , F3 , …

15
Dynamic Programming Solution for Fibonacci

 Fib(n):
 F[0] := 0; F[1] := 1;
 for i := 2 to n do
 F[i] := F[i-1] + F[i-2]
 return F[n]
 Can perform application-specific optimizations
 e.g., save space by only keeping last two numbers
computed

16
 Longest Common Subsequence

17
 For example, Find subsequences of “abcdefg”.

 “abc”, “abg”, “bdf”, “aeg”, „”acefg”, .. etc are subsequences

of
 a string of length n has 2n-1 different possible
subsequences since we do not consider the subsequence
with length 0. This implies that the time complexity of the
brute force approach will be O(n * 2n)

18
 Subsequence
 Let us consider a sequence S = <s1, s2, s3, s4, …,sn>.
 A sequence Z = <z1, z2, z3, z4, …,zm> over S is called a
subsequence of S, if and only if it can be derived from S
deletion of some elements.

19
Common Subsequence

 Suppose, X and Y are two sequences over a finite set of

elements. We can say that Z is a common subsequence
of X and Y, if Z is a subsequence of both X and Y.

20
Longest Common Subsequence

 If a set of sequences are given, the longest common

subsequence problem is to find a common subsequence
of all the sequences that is of maximal length.
 The longest common subsequence problem is a classic
computer science problem used in many application such
by revision control systems

21
 Substring Vs. Subsequence

 A substrings are consecutive parts of a string, while

subsequences need not be.

 Hence, a substring of a string is always a subsequence of

the string, but a subsequence of a string is not always a
substring of the string.

Subsequence need not be consecutive, but must be in order.

22
Example: Longest Common Subsequence

 Longest Common Subsequence

 Given two sequences x[1 . . m] and y[1 . . n], find a longest
subsequence (z) common to them both.
 Subsequence of a given sequence is the given sequence with
zero or more elements left out
 Indices are strictly increasing
x: A B C B D A B

y: B D C A B A
LCS(x, y) = BCBA or BCAB
23
Example: Longest Common Subsequence

Brute force algorithm would compare each subsequence of X

with the symbols in Y A,B,C,D, AB,
AC, AD ,
……….
1-Compare X to Y symbols ABC,ABCB,AB
CBD,………….

x: A B C B D A B

y: B D C A B A
LCS(x, y) = BCAB
24
Example: Longest Common Subsequence

Brute force algorithm would compare each subsequence of X

with the symbols in Y

1-Compare Y to X symbols

x: A B C B D A B

y: B D C A B A
LCS(x, y) = BCBA
25
 Applications
 Subsequences have applications in Bioinformatics, where computers
are used to compare, analyze, and store DNA strands.

 Take two strands of DNA, say :

 ORG1=ACGGTGTCGTGCTATGCTGATGCTGACTTATATGCTA
ORG2=CGTTCGGCTATCGTACGTTCTATTCTATGATTTCTAA

 Subsequences are used to determine how similar the two

strands of DNA are, using the DNA
bases: adenine, guanine, cytosine and thymine.

26
 LCS Algorithm

 Brute-force algorithm: 2m subsequences of x, each is

checked against y (size n) to find if it is a subsequence 
O(n 2m)

 Exponential!! Can we do any better?

27
 Another procedure to
improve number of
comparisons

28
 Define a function LCS(X, Y): gives the longest subsequences
common to X and Y. That function relies on the following
two properties:

1-If the two sequences both end in the same element

2-If the two sequences don‟t end in the same symbol.

29
Case 1: If the two sequences both end in the same element

 To find their LCS, shorten each sequence by removing the last

element, find the LCS of the shortened sequences, and to that
LCS append the removed element.

 Example: The sequences (BANANA), (ATANA).

 Remove the last element repeatedly  ANA
 Common subsequence of remaining sequence  A
 Append removed elements: LCS = AANA

30
Case 2: If the two sequences don‟t end in the same symbol.

 the LCS of X and Y is the longer of the two sequences LCS(Xi,Yj-1) and
LCS(Xi-1,Yj).

 Example: X = ABCDEFG , Y= BCDGK

 The last character of the LCS of these two sequences either ends with a
G (1) or doesn't (2).

 Case1: LCS(Xi,Yj) = LCS(Xi,Yj-1)

 Case 2: LCS(Xi,Yj) = LCS(Xi-1,Yj)

31
Overlapping subproblems

 To find LCS(X,Y) you need to find LCS(Xi,Yj-1)

and LCS(Xi-1,Yj)

 Each of these subproblems has the subproblem of

finding an (Xi-1,Yj-1).

 Smaller subproblems also share other subproblems

32
 LCS recursive solution

 Let‟s first try to find the length c (not the actual letters) of the
LCS(X,Y)
 We start with i = j = 0 (empty substrings of x and y)
 Since X0 and Y0 are empty strings, their LCS is always empty (i.e.
c[0,0] = 0)
 LCS of empty string and any other string is empty, so for every i
and j: c[0, j] = c[i,0] = 0

33
 LCS recursive solution

 0 if i  0 or j  0


C[i, j ]  c[i  1, j  1]  1 if x[i ]  y[ j ],
 max(c[i, j  1], c[i  1, j ]) otherwise


  if i  0 or j  0


LCS [ Xi, Yj ]   LCS ( X i 1 , Y j 1 ), xi if xi  y j ,

longest ( LCS ( X i , Y j 1 ), LCS ( X i 1 , Y j )) if xi  y j

34
 LCS : Algorithm

35
LCS-Length(X,Y)
1. m = length(X) // get the # of symbols in X
2. n = length(Y) // get the # of symbols in Y
Let b[1..m, 1..n] and c[1..m, 0..n] be new tables
3. for i = 1 to m
c[i,0] = 0 // special case:Y0
4. for j = 1 to n
c[0,j] = 0 // special case: X0
5. for i = 1 to m // for all Xi
6. for j = 1 to n // for all Yj
7. if ( Xi == Yj ) {
8. c[i,j] = c[i-1,j-1] + 1
8. b[i,j] =“ ” }
9. else { c[i,j] = max( c[i-1,j], c[i,j-1] )
Print either “ “ or “ ” }
10. return c 36
LCS Example
We‟ll see how LCS algorithm works on the following
example:
 X = ABCB
 Y = BDCAB

What is the Longest Common Subsequence

of X and Y?
LCS(X,Y) = BCB
X =A B C B
Y = B D CA B

37
 ABCB
LCS Example (0) BDCAB
j 0 1 2 3 4
i 5 Yj B D C A B

0 Xi

A
1

2 B

3 C

4 B

X = ABCB; m = |X| = 4
Y = BDCAB; n = |Y| = 5
Allocate array c[5,4] – from 0-5, from 0-4
38
LCS Example (1) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0

2 B
0

3 C 0

4 B 0

for i = 1 to m c[i,0] = 0
for j = 1 to n c[0,j] = 0
39
LCS Example (2) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

40
LCS Example (3) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

41
LCS Example (4) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

42
LCS Example (5) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

43
LCS Example (6) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

44
LCS Example (7) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

45
LCS Example (8) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

46
LCS Example (9) ABCB
j 0 1 2 3 4 BDCAB
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

47
LCS Example ABCB
(10) BDCAB
j 0 1 2 3 4
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

48
LCS Example ABCB
(11) BDCAB
j 0 1 2 3 4
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

49
LCS Example ABCB
(12) BDCAB
j 0 1 2 3 4
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

50
LCS Example ABCB
(13) BDCAB
j 0 1 2 3 4
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

51
LCS Example ABCB
(14) BDCAB
j 0 1 2 3 4
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

52
LCS Example ABCB
(15) BDCAB
j 0 1 2 3 4
i 5 Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2 3
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

53
 LCS Algorithm Running Time

 LCS algorithm calculates the values of each entry of the

array c[m,n]
 So what is the running time?

O(m*n)
since each c[i,j] is calculated in
constant time, and there are m*n
elements in the array

54
 How to find actual LCS
 So far, we have just found the length of LCS, but not
LCS itself.
 We want to modify this algorithm to make it output
Longest Common Subsequence of X and Y
Each c[i,j] depends on c[i-1,j] and c[i,j-1]
or c[i-1, j-1]
For each c[i,j] we can say how it was acquired:

2 2
For example, here
2 3
c[i,j] = c[i-1,j-1] +1 = 2+1=3
55
 How to find actual LCS - continued
 Remember that
c[i  1, j  1]  1 if x[i ]  y[ j ],
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise

 We can start from c[m,n] and go backwards

 Whenever c[i,j] = c[i-1, j-1]+1, remember x[i]
(because x[i] is a part of LCS)
 When i=0 or j=0 (i.e. we reached the beginning),
output remembered letters in reverse order

56
 Finding LCS
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2 3

57
 Finding LCS (2)
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2 3
LCS (reversed order ): B C B
LCS (straight order): B C B
58 A palindrome!
 Your Turn
Find the LCS for input Sequences
x = XMJYAUZ and y = MZJAWXU

LCS= MJAU and length=4

59
 Your Turn
Find the LCS for input Sequences
“AGGTAB” and “GXTXAYB”

60
Examples:
LCS for input Sequences “ABCDGH” and
“AEDFHR” is “ADH” of length 3.
LCS for input Sequences “AGGTAB” and
“GXTXAYB” is “GTAB” of length 4.

61
Your Turn
Longest Common Subsequence

springtime ncaa tournament basketball

printing north carolina krzyzewski

62
 Thank You

63