AA02 Recurrences

A N A LY T I C C O M B I N AT O R I C S
PART ONE
2. Recurrences
http://aofa.cs.princeton.edu
PART ONE
2. Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
2a.Recur.Values
What is a recurrence?
Def. A recurrence is an equation that recursively defines a sequence.
Familiar example 1: Fibonacci numbers
recurrence
= − + − ≥ = =
MUST specify
for all N with
sequence
initial conditions
0, 1, 1, 2, 3, 5, 8, 13, 21, ...
Q. Simple formula for sequence (function of N )?
3
What is a recurrence?
Recurrences directly model costs in programs.
Familiar example 2: Quicksort (see lecture 1)
recurrence program
public class Quick
= + + ( + ) {
private static int partition(Comparable[] a, int lo, int hi)
{
int i = lo, j = hi+1;
while (true)
≥ = {
while (less(a[++i], a[lo])) if (i == hi) break;
while (less(a[lo], a[--j])) if (j == lo) break;
if (i >= j) break;
sequence exch(a, i, j);
}
exch(a, lo, j);
0, 2, 5, 8 2/3, 12 5/6, 17 2/5,... return j;
}
private static void sort(Comparable[] a, int lo, int hi)

{
if (hi <= lo) return;
int j = partition(a, lo, hi);
sort(a, lo, j-1);
sort(a, j+1, hi);
}
}
4
Common-sense rule for solving any recurrence
Use your computer to compute values. = − + − ≥ = =
public static void F(int N)

{
Use a recursive program? if (N == 0) return 0; ✘
if (N == 1) return 1;
return F(N-1) + F(N-2);
} F(50)
F(49) F(48)
NO, NO, NO: Takes exponential time! F(48) F(47) F(47) F(46)
F(47) F(46) F(46) F(45) F(46) F(45) F(45) F(44)
Top of recursion tree for naïve Fibonacci function
long[] F = new long[51];

F[0] = 0; F[1] = 1;
Instead, save all values in an array.
if (N == 1) return 1; ✓
for (int i = 2; i <= 50; i++)
F[i] = F[i-1] + F[i-2];
5
Common-sense starting point for solving any recurrence
Use your computer to compute initial values.
First step: Download "standard model" from Algorithms, 4th edition booksite.
StdIn Standard Input
StdOut Standard Output
StdDraw Standard Drawings
StdRandom Random Numbers
... (Several other libraries)
http://algs4.cs.princeton.edu
6
Use your computer to compute initial values (modern approach).

Ex. 1: Fibonacci = − + − = = Sequence.java
public interface Sequence
Fib.java public class Fib implements Sequence {
{ public double eval(int N);
private final double[] F;
}
public Fib(int maxN)
{ Compute all values % java Fib 15
F = new double[maxN+1]; in the constructor 0.0
F[0] = 0; F[1]= 1; 1.0
for (int N = 2; N <= maxN; N++) 1.0
F[N] = F[N-1] + F[N-2];
2.0
}
3.0
public double eval(int N) 5.0
{ return F[N]; } 8.0
13.0
public static void main(String[] args) 21.0
{ 34.0
int maxN = Integer.parseInt(args[0]); 55.0
Fib F = new Fib(maxN); 89.0
for (int i = 0; i < maxN; i++) 144.0
StdOut.println(F.eval(i)); 233.0
} 377.0
}
7
Ex. 2: Quicksort =( + ) +
QuickSeq.java
public class QuickSeq implements Sequence
{
private final double[] c; % java QuickSeq 15
0.000000
public QuickSeq(int maxN) 2.000000
{ 5.000000
c = new double[maxN+1]; 8.666667
c[0] = 0; 12.833333
for (int N = 1; N <= maxN; N++) 17.400000
c[N] = (N+1)*c[N-1]/N + 2; 22.300000
} 27.485714
32.921429
{ return c[N]; } 44.437302
50.477056
{ 63.043745
// Similar to Fib.java. 69.546870
}
}
8
Use your computer to plot initial values. QuickSeq.java

public class QuickSeq implements Sequence
{
// Implementation as above.
Values.java
public class Values public static void main(String[] args)
{ {
public static void show(Sequence f, int maxN) int maxN = Integer.parseInt(args[0]);
{ QuickSeq q = new QuickSeq(maxN);
double max = 0; Values.show(q, maxN);
for (int N = 0; N < maxN; N++) }
}
if (f.eval(N)>max) max = f.eval(N);
for (int N = 0; N < maxN; N++)
{
double x = 1.0*N/maxN;
double y = 1.0*f.eval(N)/max;
StdDraw.filledCircle(x, y, .002);
}
StdDraw.show();
}
}
% java QuickSeq 1000
9
PART ONE
2. Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2a.Recur.Values
PART ONE
2. Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2b.Recur.Telescope
Telescoping a (linear first-order) recurrence
Linear first-order recurrences telescope to a sum.
Example 1.
= + =
Apply equation for n−1 = +( )+
Do it again = +( )+( )+
Continue, leaving a sum = +
( + )
Evaluate sum = Check.
( + ) ( )
= +
Challenge: Need to be able to evaluate the sum.

12
Elementary discrete sums
geometric series =
<
( )
arithmetic series = =
<
+
binomial (upper) =
+
see Knuth volume 1
for many more
binomial theorem =( + )
Harmonic numbers =
+
Vandermonde convolution =
13
Telescoping a (linear first-order) recurrence (continued)
When coefficients are not 1, multiply/divide by a summation factor.
Example 2.
= + =
Divide by 2n = +
Telescope to a sum = =
=
Check.
= ( ) +
Challenge: How do we find the summation factor?
14
Telescoping a (linear first-order) recurrence (continued)
Q. What’s the summation factor for = + ... ?

A. Divide by ...
Example 3.
= + + > =
summation factor:
+
... = +
Divide by n+1 = +
+ +
2
Telescope = = +
+
= ( + )( + )
Challenge: Still need to be able to evaluate sums.

15
In-class exercise 1.
Verify the solution for Example 3.
Check initial values
= + + > = = ( + )( + )
= + = = ( )=
= + = = ( )=
= ( )
= + = /
= ( / + / + / )
= /
Proof
+ ( )+ = ( + )( )+
= ( + )( + )
16
In-class exercise 2.
Solve this recurrence:
=( ) + > =
Hard way:
summation factor: ··· =

( )
Easy way: = =
=
WHY?
17
PART ONE
Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2b.Recur.Telescope
PART ONE
Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2c.Recur.Types
Types of recurrences
linear an = nan 1 1
first
order
nonlinear an = 1/(1 + an 1)
linear an = an 1 + 2an 2
p
second order nonlinear an = an 1 an 2+ an 2
variable
coefficients an = nan 1 + (n 1)an 2 +1
higher order an = f (an 1 , an 2 , . . . , a n t )
full history an = n + an 1 + an 2 ... + a1
divide-and-conquer an = abn/2c + adn/2e + n
20
Nonlinear first-order recurrences
! "
Example. (Newton’s method ) = + [Typical in scientific computing]
−
−
SqrtTwo.java
public class SqrtTwo implements Sequence
{
private final double[] c;
% java SqrtTwo 10
1.0
public SqrtTwo(int maxN)
{
1.5
c = new double[maxN+1]; 1.4166666666666665
c[0] = 1; quadratic convergence:
number of significant 1.4142156862745097
for (int N = 1; N <= maxN; N++)
c[N] = (c[N-1] + 2/c[N-1])/2; digits doubles for 1.4142135623746899
} each iteration 1.414213562373095
{ return c[N]; } 1.414213562373095
{ 1.414213562373095
int maxN = Integer.parseInt(args[0]);
SqrtTwo test = new SqrtTwo(maxN);
for (int i = 0; i < maxN; i++)
StdOut.println(test.eval(i));
}
}
21
Higher-order linear recurrences
[ Stay tuned for systematic solution using generating functions (next lecture) ]
Example 4.
= = =
Postulate that an = xn =
Divide by xn−2 + =
Factor ( )( )=
Form of solution must be = +
Use initial conditions to = = + Note dependence

solve for coeﬃcients on initial conditions
= = +
Solution is c0 = 1 and c1 = −1 =
22
Higher-order linear recurrences
[ Stay tuned for systematic solution using generating functions (next lecture) ]
Example 5. Fibonacci numbers

= + = =
Postulate that an = xn = +
Divide by xn−2 =
+
ˆ) = =
Factor ( )(
ˆ=
Form of solution must be = + ˆ
Use initial conditions to = = + Note dependence

solve for coeﬃcients on initial conditions
= = +ˆ
ˆ
Solution =
23
Higher-order linear recurrences (continued)
Procedure amounts to an algorithm.
Multiple roots? Add nαn terms (see text)
Need to compute roots? Use symbolic math package.
sage: realpoly.<z> = PolynomialRing(CC)

sage: factor(z^2-z-1)
(z - 1.61803398874989) * (z + 0.618033988749895)
Complex roots? Stay tuned for systematic solution using GFs (next lecture)
24
Divide-and-conquer recurrences
Divide and conquer is an effective technique in algorithm design.
Recursive programs map directly to recurrences.
Classic examples:
• Binary search
• Mergesort
• Batcher network
• Karatsuba multiplication
• Strassen matrix multiplication
25
PART ONE
Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2c.Recur.Types
PART ONE
Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2d.Recur.Mergesort
Warmup: binary search
Everyone’s first divide-and-conquer algorithm
// Precondition: array a[] is sorted.

public static int rank(int key, int[] a)
{
int lo = 0;
int hi = a.length - 1;
while (lo <= hi)
{
// Key is in a[lo..hi] or not present.
int mid = lo + (hi - lo) / 2;
if (key < a[mid]) hi = mid - 1;
else if (key > a[mid]) lo = mid + 1;
else return mid;
}
return -1;
}
Number of compares in the worst case

= / + > =
28
Analysis of binary search (easy case)
= / + > =
Exact solution for N = 2n .
= + > =
a0 +
+1
Telescope to a sum = =
+1
=
Check. = ( / )+
29
Analysis of binary search (general case)
Easy by correspondence with binary numbers
Define BN to be the number of bits in the binary representation of N. Example.
• B1 = 1. 1101011 110101 1
• Removing the rightmost bit of N gives N/2 . 107 53

Therefore = / + > =
N N/2
same recurrence as for binary search
Theorem. BN = lg N +1 +
= + <
Proof. Immediate by definition of . < + = =
N 1 2 3 4 5 6 7 8 9
binary 1 10 11 100 101 110 111 1000 1001
lg N 0 1.0 1.58... 2.0 2.32... 2.58... 2.80... 3 3.16...
lg N 0 1 1 2 2 2 2 3 3
lg N +1 1 2 2 3 3 3 3 4 4
30
Mergesort
Everyone’s second divide-and-conquer algorithm
public class Merge

{
...
private static void
sort(Comparable[] a, Comparable[] aux, int lo, int hi)
{
if (hi <= lo) return;
int mid = lo + (hi - lo) / 2;
sort(a, aux, lo, mid);
sort(a, aux, mid + 1, hi);
merge(a, aux, lo, mid, hi);
}
...
}
For simplicity, assume merge implementation uses N compares
Number of compares for sort: = / + / + > = 0
31
Analysis of mergesort (easy case)
Already solved for N = 2n
Solution: =
32
Analysis of mergesort (general case)
Solution: =
Q. For quicksort, the number of compares is ( )
Is the number of compares for mergesort + for some constant α?
A. NO !
33
Coefficient of the linear term for mergesort
public class MergeLinearTerm implements Sequence

{
private final double[] c;
public MergeLinear(int maxN)

{
c = new double[maxN+1]; % java MergeLinearTerm 512
c[0] = 0;
c[N] = N + c[N/2] + c[N-(N/2)];
c[N] -= N*Math.log(N)/Math.log(2)) + N;
}
public double eval(int N)

{ return c[N]; }
public static void main(String[] args)

{
int maxN = Integer.parseInt(args[0]);
MergeLinearTerm M = new MergeLinearTerm(maxN);
Values.show(M, maxN);
}
}
34
Analysis of mergesort (general case)
/ = ( + )/
Same formula for N+1. + = ( + )/ + ( + )/ + +
= / + / + + + 1 0 1 1 1
2 1 1 1 2
3 1 2 2 2
Subtract. + = / + / + 4 2 2 2 3
5 2 3 3 3
6 3 3 3 4
Define DN = CN+1 - CN. = / + 7 3 4 4 4
different 8 4 4 4 5
initial 9 4 5 5 5
Solve as for binary search. = lg + value
/ + = ( + )/
Telescope.
= + ( lg + )
<
Theorem. CN = N −1 + number of bits in binary representation of numbers < N
35
Combinatorial correspondence
SN = number of bits in the binary rep. of all numbers < N
/2 /2 1
1 1 1 1
10 10 10 10
11 11 11 11
100 100 100 100
101 101 101 101
110 110 110 110
111 111 111 111
1000 = 1000 + 1000 + 1000
1001 1001 1001 1001
1010 1010 1010 1010
1011 1011 1011 1011
1100 1100 1100 1100
1101 1101 1101 1101
1110 1110 1110 1110
= /2 + /2 + 1
Same recurrence as mergesort (except for −1): = + 1

36
Number of bits in all numbers < N (alternate view)
842 1
bits are in an subtract off red 0s
N by lgN+1 box column by column
0000
0001
= ( + 1) 2
0010
0
0011
0100 +1
= 2 + +1
0101
0110
0111
1000
1001
= + 1
1010 +1
= 2 +2
1011
1100
1101
1110 Theorem. Number of compares for mergesort is 2 +1
+2
+1 37
Analysis of mergesort (summary)
Solution: =
+1
Theorem. Number of compares for mergesort is 2 +2
Alternate formulation (Knuth). = lg + ( )
Notation: lg = lg {lg }
( )
{lg }
+
{lg }
=
( ) {lg } {lg }
✓
38
PART ONE
Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2d.Recur.Mergesort
PART ONE
Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2e.Recur.Master
Divide-and-conquer algorithms
Suppose that an algorithm attacks a problem of size N by

• Dividing into α parts of size about N/β.
• Solving recursively.
• Combining solutions with extra cost Θ(N γ(log N )δ)
only valid when
N is a power of 2
Example 1 (mergesort): α = 2, β = 2, γ = 1, δ = 0 = / +
Example 2 (Batcher network): α = 2, β = 2, γ = 1, δ = 1 = / + lg
Example 3 (Karatsuba multiplication): α = 3, β = 2, γ = 1, δ = 0 = / +
Example 4 (Strassen matrix multiply): α = 7, β = 2, γ = 1, δ = 0 = / +
41
“Master Theorem” for divide-and-conquer algorithms
Suppose that an algorithm attacks a problem of size n by
dividing into α parts of size about n/β with extra cost Θ(nγ(log n)δ)
Theorem. The solution to the recurrence

= / + ( ) + / + ( ) + ... + / + ( ) + ( (log ) )
α terms >
is given by
= ( (log ) ) < log
+
= ( (log ) ) = log
log
= ( ) > log AN INTRODUCTION
ANALYSIS
TO THE
ALGORITHMS
OF
S E C O N D E D I T I O N
< ROBERT SEDGEWICK
Example: α = 3
PHILIPPE FLAJOLET
β=2
β=3
β=4
42
Typical “Master Theorem” applications
Master Theorem
= ( (log ) < log
+
= ( (log ) = log
log
= ( ) > log
Asymptotic growth rate
Example 1 (mergesort): α = 2, β = 2, γ = 1, δ = 0 Θ(N log N )
Example 2 (Batcher network): α = 2, β = 2, γ = 1, δ = 1 Θ(N (log N )2 )
Example 3 (Karatsuba multiplication): α = 3, β = 2, γ = 1, δ = 0 Θ(N log2 3 ) = Θ(N 1.585... )
Example 4 (Strassen matrix multiply): α = 7, β = 2, γ = 1, δ = 0 Θ(N log2 7 ) = Θ(N 2.807... )
43
Versions of the “Master Theorem”

1. Precise results are available for certain applications AN INTRODUCTION
ANALYSIS
TO THE
in the analysis of algorithms. ALGORITHMS

OF
ROBERT SEDGEWICK
PHILIPPE FLAJOLET
2. General results are available for proofs

in the theory of algorithms.
3. A full solution using analytic combinatorics

was derived in 2011 by Szpankowski and Drmota.
see “A Master Theorem for Divide-and-Conquer Recurrences”

by Szpankowski and Drmota (SODA 2011).
44
PART ONE
Recurrences
• Telescoping
OF
• Mergesort
• Master Theorem
2e.Recur.Master
Exercise 2.17
Percentage of three nodes at the bottom level of a 2-3 tree?
Typical 2-3 tree built from random keys
AN INTRODUCTION
ANALYSIS
TO THE
ALGORITHMS
OF
ROBERT SEDGEWICK
PHILIPPE FLAJOLET
46
Exercise 2.69
Details of divide-by-three and conquer?
AN INTRODUCTION
ANALYSIS
TO THE
ALGORITHMS
OF
ROBERT SEDGEWICK
PHILIPPE FLAJOLET
47
Assignments for next lecture
1. Read pages 41-86 in text.
AN INTRODUCTION
ANALYSIS
TO THE
ALGORITHMS
OF
2. Write up solution to Ex. 2.17.

ROBERT SEDGEWICK
PHILIPPE FLAJOLET
3. Set up StdDraw from Algs booksite
4. Do Exercise 2.69.
48
PART ONE
2. Recurrences

AA02 Recurrences

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AA02 Recurrences

Uploaded by

Copyright:

Available Formats

A N A LY T I C C O M B I N AT O R I C S

Def. A recurrence is an equation that recursively defines a sequence.

Familiar example 1: Fibonacci numbers

Q. Simple formula for sequence (function of N )?

Recurrences directly model costs in programs.

Familiar example 2: Quicksort (see lecture 1)

private static void sort(Comparable[] a, int lo, int hi)

Use your computer to compute values. = − + − ≥ = =

public static void F(int N)

F(47) F(46) F(46) F(45) F(46) F(45) F(45) F(44)

Top of recursion tree for naïve Fibonacci function

long[] F = new long[51];

Use your computer to compute initial values.

StdIn Standard Input

StdOut Standard Output

StdDraw Standard Drawings

StdRandom Random Numbers

... (Several other libraries)

Use your computer to compute initial values (modern approach).

Use your computer to plot initial values. QuickSeq.java

% java QuickSeq 1000

Apply equation for n−1 = +( )+

Continue, leaving a sum = +

Challenge: Need to be able to evaluate the sum.

When coefficients are not 1, multiply/divide by a summation factor.

Q. What’s the summation factor for = + ... ?

Challenge: Still need to be able to evaluate sums.

Verify the solution for Example 3.

Check initial values

Solve this recurrence:

summation factor: ··· =

higher order an = f (an 1 , an 2 , . . . , a n t )

full history an = n + an 1 + an 2 ... + a1

divide-and-conquer an = abn/2c + adn/2e + n

Form of solution must be = +

Use initial conditions to = = + Note dependence

Example 5. Fibonacci numbers

Use initial conditions to = = + Note dependence

Procedure amounts to an algorithm.

Multiple roots? Add nαn terms (see text)

Need to compute roots? Use symbolic math package.

sage: realpoly.<z> = PolynomialRing(CC)

Divide and conquer is an effective technique in algorithm design.

Recursive programs map directly to recurrences.

// Precondition: array a[] is sorted.

Number of compares in the worst case

Exact solution for N = 2n .

Easy by correspondence with binary numbers

Define BN to be the number of bits in the binary representation of N. Example.

• Removing the rightmost bit of N gives N/2 . 107 53

public class Merge

For simplicity, assume merge implementation uses N compares

Number of compares for sort: = / + / + > = 0

Number of compares for sort: = / + / + > = 0

Already solved for N = 2n

Number of compares for sort: = / + / + > = 0

Q. For quicksort, the number of compares is ( )

Is the number of compares for mergesort + for some constant α?

public class MergeLinearTerm implements Sequence

public MergeLinear(int maxN)

public double eval(int N)

public static void main(String[] args)

Number of compares for sort: = / + / + > = 0