Algorithms:

COMP3121/9101

Aleks Ignjatović, ignjat@cse.unsw.edu.au

office: 504 (CSE building K 17)
Admin: Song Fang, cs3121@cse.unsw.edu.au

School of Computer Science and Engineering

University of New South Wales Sydney

2. DIVIDE-AND-CONQUER

COMP3121/3821/9101/9801 1 / 52
An old puzzle

Problem
We are given 27 coins of the same denomination; we know that one of
them is counterfeit and that it is lighter than the others. Find the
counterfeit coin by weighing coins on a pan balance only three times.

Hint
You can reduce the search space by a third in one weighing!

COMP3121/3821/9101/9801 2 / 52
An old puzzle

Solution
Divide the coins into three groups of nine, say A, B and C.
Weigh group A against group B.
If one group is lighter than the other, it contains the counterfeit
coin.
If instead both groups have equal weight, then group C contains the
counterfeit coin!
Repeat with three groups of three, then three groups of one.

COMP3121/3821/9101/9801 3 / 52
Divide and Conquer

This method is called “divide-and-conquer”.

We have already seen a prototypical “serious” algorithm designed
using such a method: merge sort.
We split the array into two, sort the two parts recursively and
then merge the two sorted arrays.
We now look at a closely related but more interesting problem of
counting inversions in an array.

COMP3121/3821/9101/9801 4 / 52
Counting the number of inversions

Assume that you have m users ranking the same set of n movies.
You want to determine for any two users A and B how similar
their tastes are (for example, in order to make a recommender
system).

How should we measure the degree of similarity of two users A and

B?

Lets enumerate the movies on the ranking list of user B by

assigning to the top choice of user B index 1, assign to his second
choice index 2 and so on.

For the ith movie on B 0 s list we can now look at the position (i.e.,
index) of that movie on A0 s list, denoted by a(i).

COMP3121/3821/9101/9801 5 / 52
Counting the number of inversions

A good measure of how different these two users are, is the total
number of inversions, i.e., total number of pairs of movies i, j such
that movie i precedes movie j on B 0 s list but movie j is higher up
on A0 s list than the movie i.

In other words, we count the number of pairs of movies i, j such

that i < j (movie i precedes movie j on B 0 s list) but a(i) > a(j)
(movie i is in the position a(i) on A0 s list which is after the
position a(j) of movie j on A0 s list.

COMP3121/3821/9101/9801 6 / 52
Counting the number of inversions

B 1 2 3 4 5 6 7 8 9 10 11 12

A 1 5 9 12 7 10 3 4 6 8 2 11

a(9) = 3 a(4) = 8

a(7) = 5 a(2) = 11

COMP3121/3821/9101/9801 7 / 52
Counting the number of inversions

For example 1 and 2 do not form an inversion because a(1) < a(2)
(a(1) = 1 and a(2) = 11 because a(1) is on the first and a(2) is on
the 11th place in A);
However, for example 4 and 7 do form an inversion because
a(7) < a(4) (a(7) = 5 because seven is on the fifth place in A and
a(4) = 8)

COMP3121/3821/9101/9801 8 / 52
Counting the number of inversions

An easy way to count the total number of inversions between two

lists is by looking at all pairs i < j of movies on one list and
determining if they are inverted in the second list, but this would
produce a quadratic time algorithm, T (n) = Θ(n2 ).

We now show that this can be done in a much more efficient way,
in time O(n log n), by applying a Divide-And-Conquer strategy.

Clearly, since the total number of pairs is quadratic in n, we

cannot afford to inspect all possible pairs.

The main idea is to tweak the Merge-Sort algorithm, by

extending it to recursively both sort an array A and determine
the number of inversions in A.

COMP3121/3821/9101/9801 9 / 52
Counting the number of inversions
We split the array A into two (approximately) equal parts
Atop = A[1 . . . bn/2c] and Abottom = A[bn/2c + 1 . . . n].
Note that the total number of inversions in array A is equal to the
sum of the number of inversions I(Atop ) in Atop (such as 9 and 7)
plus the number of inversions I(Abottom ) in Abottom (such as 4 and
2) plus the number of inversions I(Atop , Abottom ) across the two
halves (such as 7 and 4).

1 9 7 3 4 6 8 2 5 A

Atop Abottom

COMP3121/3821/9101/9801 10 / 52
Counting the number of inversions
We now recursively sort arrays Atop and Abottom also obtaining in the process
the number of inversions I(Atop ) in the sub-array Atop and the number of
inversions I(Abottom ) in the sub-array Abottom .
We now merge the two sorted arrays Atop and Abottom while counting the
number of inversions I(Atop , Abottom ) which are across the two sub-arrays.
When the next smallest element among all elements in both arrays is an
element in Abottom , such an element clearly is in an inversion with all the
remaining elements in Atop and we add the total number of elements remaining
in Atop to the current value of the number of inversions across Atop and
Abottom .

1 11 9 12 7 10 3 4 6 8 2 5 A
a1=1 a9=3 a7=5 a4=8 a2=11

Atop Abottom
Atop
1 7 9 10 11 12

1 Abottom
2 33 4 5 6 8

COMP3121/3821/9101/9801 11 / 52
Counting the number of inversions

Whenever the next smallest element among all elements in both

arrays is an element in Atop , such an element clearly is not involved
in any inversions across the two arrays (such as 1, for example).

After the merging operation is completed, we obtain the total

number of inversions I(Atop , Abottom ) across Atop and Abottom .

The total number of inversions I(A) in array A is finally obtained

as:
I(A) = I(Atop ) + I(Abottom ) + I(Atop , Abottom )

Next: we study applications of divide and conquer to arithmetic

of very large integers.

COMP3121/3821/9101/9801 12 / 52
Basics revisited: how do we add two numbers?

C C C C C carry
X X X X X first integer
+ X X X X X second integer
-----------
X X X X X X result

adding 3 bits can be done in constant time;

the whole algorithm runs in linear time i.e., O(n) many steps.

can we do it faster than in linear time?

no, because we have to read every bit of the input
no asymptotically faster algorithm

COMP3121/3821/9101/9801 13 / 52
Basics revisited: how do we multiply two numbers?

X X X X <- first input integer

* X X X X <- second input integer
-------
X X X X \
X X X X \ O(n^2) intermediate operations:
X X X X / O(n^2) elementary multiplications
X X X X / + O(n^2) elementary additions
---------------
X X X X X X X X <- result of length 2n

We assume that two X’s can be multiplied in O(1). time (each X

could be a bit or a digit in some other base).
Thus the above procedure runs in time O(n2 ).
Can we do it in LINEAR time, like addition?
No one knows!
“Simple” problems can actually turn out to be difficult!
COMP3121/3821/9101/9801 14 / 52
Can we do multiplication faster than O(n2 )?

Let us try a divide-and-conquer algorithm:

take our two input numbers A and B, and split them into two halves:
n
A = A1 2 2 + A0 XX . . . X} XX
| {z | {z. . . X}
n n n
B = B1 2 + B0
2
2 2
• A0 , B0 - the least significant bits; A1 , B1 the most significant bits.
• AB can now be calculated as follows:
n
AB = A1 B1 2n + (A1 B0 + B1 A0 )2 2 + A0 B0 (1)

What we mean is that the product AB can be calculated recursively by

the following program:

COMP3121/3821/9101/9801 15 / 52
 1: function Mult(A, B)
2: if |A| = |B| = 1 then return AB
3: else
4: A1 ← MoreSignificantPart(A);
5: A0 ← LessSignificantPart(A);
6: B1 ← MoreSignificantPart(B);
7: B0 ← LessSignificantPart(B);
8: X ← Mult(A0 , B0 );
9: Y ← Mult(A0 , B1 );
10: Z ← Mult(A1 , B0 );
11: W ← Mult(A1 , B1 );
12: return W 2n + (Y + Z) 2n/2 + X
13: end if
14: end function

COMP3121/3821/9101/9801 16 / 52
How many steps does this algorithm take?

Each multiplication of two n digit numbers is replaced by four

multiplications of n/2 digit numbers: A1 B1 , A1 B0 , B1 A0 , A0 B0 ,
plus we have a linear overhead to shift and add:
n
T (n) = 4T + cn (2)
2

COMP3121/3821/9101/9801 17 / 52
Can we do multiplication faster than O(n2 )?
Claim: if T (n) satisfies
n
T (n) = 4T + cn (3)
2
then

T (n) = n2 (c + 1) − c n

Proof: By “fast” induction. We assume it is true for n/2:

n  n 2 n
T = (c + 1) − c
2 2 2

and prove that it is also true for n:



2
T (n) = 4 T n2 + c n = 4 n2 (c + 1) − n


2 c + cn
= n2 (c + 1) − 2c n + c n = n2 (c + 1) − c n
COMP3121/3821/9101/9801 18 / 52
Can we do multiplication faster than O(n2 )?
n


Thus, if T (n) satisfies T (n) = 4 T 2 + c n then

T (n) = n2 (c + 1) − c n = Θ(n2 )

i.e., we gained nothing with our divide-and-conquer!

Is there a smarter multiplication algorithm taking less than O(n2 )

many steps??

Remarkably, there is, but first some history:

In 1952, one of the most famous mathematicians of the 20th century,
Andrey Kolmogorov, conjectured that you cannot multiply in less than
Ω(n2 ) elementary operations. In 1960, Karatsuba, then a 23-year-old
student, found an algorithm (later it was called “divide and conquer”)
that multiplies two n-digit numbers in Θ(nlog2 3 ) ≈ Θ(n1.58... )
elementary steps, thus disproving the conjecture!! Kolmogorov was
shocked!
COMP3121/3821/9101/9801 19 / 52
The Karatsuba trick

How did Karatsuba do it??

Take again our two input numbers A and B, and split them into two
halves:
n
A = A1 2 2 + A0 XX . . . X} XX
| {z | {z. . . X}
n n n
B = B1 2 2 + B0
2 2
• AB can now be calculated as follows:
n
AB = A1 B1 2n + (A1 B0 + A0 B1 )2 2 + A0 B0
n
= A1 B1 2n + ((A1 + A0 )(B1 + B0 ) − A1 B1 − A0 B0 )2 2 + A0 B0

• So we have saved one multiplication at each recursion round!

COMP3121/3821/9101/9801 20 / 52
• Thus, the algorithm will look like this:
1: function Mult(A, B)
2: if |A| = |B| = 1 then return AB
3: else
4: A1 ← MoreSignificantPart(A);
5: A0 ← LessSignificantPart(A);
6: B1 ← MoreSignificantPart(B);
7: B0 ← LessSignificantPart(B);
8: U ← A0 + A1 ;
9: V ← B0 + B1 ;
10: X ← Mult(A0 , B0 );
11: W ← Mult(A1 , B1 );
12: Y ← Mult(U, V);
13: return W 2n + (Y − X − W ) 2n/2 + X
14: end if
15: end function
• How fast is this algorithm?

COMP3121/3821/9101/9801 21 / 52
How many multiplications does this take? (addition is in linear time!)
We need A1 B1 , A0 B0 and (A1 + A0 )(B1 + B0 ); thus
n
T (n) = 3 T + cn
2

COMP3121/3821/9101/9801 22 / 52
The Karatsuba trick
Clearly, the run time T (n) satisfies the recurrence
 
n
T (n) = 3 T + cn
2
and this implies (by replacing n with n/2)
   
n n n
T = 3T +c
2 22 2
and by replacing n with n/22    
n n n
T = 3T +c
22 23 22

 ... 
   
n n n
So we get T (n) = 3 T +c n = 3 3T

+ c  + cn
2 22 2
| {z } | {z }

 
  
2 n 3n 2 n n 3n
=3 T +c + c n = 3 3T +c 2+c + cn
22 2 23 2 2
| {z } | {z }

 
32 n 32 n
   
3 n 3n 3 n n 3n
=3 T 3
+c 2 + c + c n = 3 3T 4
+c 3+c 2 +c + cn = ...
2 2 2 2 2 2 2
| {z } | {z }
COMP3121/3821/9101/9801 23 / 52
The Karatsuba trick

       
n n n n
T (n) = 3 T +c n = 3 3T 2
+c +c n = 32 T +c 3n
2 + cn
2 2 2 22
| {z } | {z } | {z }
 
 
2 n n 3
  2
= 3 3T + c 2  + c 3n
2 + cn = 3 T
n
23
+ c 322n + c 3n
2 + cn
23 2
| {z }
 
n  2 
= 33 T +c n 322 + 23 + 1
23
| {z }
 
 
3 n n  2 
= 3 3T 4
+ c 3  + c n 322 + 32 + 1
2 2
| {z }
   3 2

4
= 3 T 24 + c n 23 + 232 + 32 + 1
n 3

...
 
32
 
3 blog2 nc−1
= 3blog2 nc T n 3


+ cn + ... + 22
+ +1
b2log2 n c 2 2

3 log2 n −1
 
 
3 log2 n
≈ 3log2 n T (1) + c n 2
= 3log2 n T (1) + 2c n


3 −1 2 −1
2

COMP3121/3821/9101/9801 24 / 52
The Karatsuba trick

So we got
 log2 n !
log2 n 3
T (n) ≈ 3 T (1) + 2c n −1
2

We now use alogb n = nlogb a to get:

 
log 3 log2 32
− 1 = nlog2 3 T (1)+2c n nlog2 3−1 − 1


T (n) ≈ n 2 T (1)+2c n n
= nlog2 3 T (1) + 2c nlog2 3 − 2c n
= O(nlog2 3 ) = O(n1.58... ) <
< n2

Please review the basic properties of logarithms and the asymptotic

notation from the review material (the first item at the class webpage
under “class resources”.)

COMP3121/3821/9101/9801 25 / 52
Some Important Questions:

Can we multiply large integers faster than O nlog2 3 ??

1

2 Can we avoid messy computations like:

n   n  n  n  3n
2
T (n) = 3T + cn = 3 3T +c + cn = 3 T +c + cn
2 22 2 22 2

2
  n  n  3n
3
 n  32 n 3n
= 3 3T +c +c + cn = 3 T +c +c + cn
23 22 2 23 22 2
 
 n  3 2 3
3
= 3 T + cn  + + 1 =
23 22 2
 
3
  n  n  32 3
= 3 3T +c + cn  + + 1 =
24 23 22 2
 
 n  3 3 32 3
4
= 3 T + cn  + + + 1 =
24 23 22 2

...
 
 3 blog
2 nc−1 32
!
blog2 nc n 3
= 3 T + cn  + ... + + + 1
b2log2 n c 2 22 2

3 log2 n − 1
 
log2 n 2
≈ 3 T (1) + cn
3 −1
2
 3 log n !
log2 n 2
= 3 T (1) + 2cn −1
2

COMP3121/3821/9101/9801 26 / 52
Recurrences

Recurrences are important to us because they arise in estimations

of time complexity of divide-and-conquer algorithms.

Merge-Sort(A, p, r) *sorting A[p..r]*

1 if p < r
2 then q ← b p+r
2 c
3 Merge-Sort(A, p, q)
4 Merge-Sort(A, q + 1, r)
5 Merge(A, p, q, r)

Since Merge(A, p, q, r) runs in linear time, the runtime T (n) of

Merge-Sort(A, p, r) satisfies
n
T (n) = 2T + cn
2
COMP3121/3821/9101/9801 27 / 52
Recurrences

Let a ≥ 1 be an integer and b > 1 a real number;

Assume that a divide-and-conquer algorithm:
reduces a problem of size n to a many problems of smaller size n/b;
the overhead cost of splitting up/combining the solutions for size
n/b into a solution for size n is if f (n),
then the time complexity of such algorithm satisfies
n
T (n) = a T + f (n)
b

Note: we should be writing

l n m
T (n) = a T + f (n)
b
but it can be shown that ignoring the integer parts and additive
constants is OK when it comes to obtaining the asymptotics.
COMP3121/3821/9101/9801 28 / 52
 n
T (n) = a T + f (n)
b

size of instance = n

a many size of instances = n/b

instances
… size of instances = n/b2
. depth of
… … . recursion:
. log ! 𝑛
…. .
. . . .
. .
. . .
. .
. .
… …
size of instances = 1

COMP3121/3821/9101/9801 29 / 52
Some recurrences can be solved explicitly, but this tends to be
tricky.

Fortunately, to estimate efficiency of an algorithm we do not need

the exact solution of a recurrence

We only need to find:

1 the growth rate of the solution i.e., its asymptotic behaviour;
2 the (approximate) sizes of the constants involved (more about
that later)

This is what the Master Theorem provides (when it is

applicable).

COMP3121/3821/9101/9801 30 / 52
Master Theorem:
Let:
a ≥ 1 be an integer and and b > 1 a real;
f (n) > 0 be a non-decreasing function;
T (n) be the solution of the recurrence T (n) = a T (n/b) + f (n);
Then:
1 If f (n) = O(nlogb a−ε ) for some ε > 0, then T (n) = Θ(nlogb a );
2 If f (n) = Θ(nlogb a ), then T (n) = Θ(nlogb a log2 n);
3 If f (n) = Ω(nlogb a+ε ) for some ε > 0, and for some c < 1 and some n0 ,

a f (n/b) ≤ c f (n)

holds for all n > n0 , then T (n) = Θ(f (n));

4 If none of these conditions hold, the Master Theorem is NOT applicable.

(But often the proof of the Master Theorem can be tweaked to obtain
the asymptotic of the solution T (n) in such a case when the Master
Theorem does not apply; an example is T (n) = 2T (n/2) + n log n).

COMP3121/3821/9101/9801 31 / 52
Master Theorem - a remark

Note that for any b > 1,

logb n = logb 2 log2 n;

Since b > 1 is constant (does not depend on n), we have for

c = logb 2 > 0

logb n = c log2 n;
1
log2 n = logb n;
c
Thus, logb n = Θ(log2 n) and also log2 n = Θ(logb n).

So whenever we have f (n) = Θ(g(n) log n) we do not have to

specify what base the log is - all bases produce equivalent
asymptotic estimates (but we do have to specify b in expressions
such as nlogb a ).
COMP3121/3821/9101/9801 32 / 52
Master Theorem - Examples

Let T (n) = 4 T (n/2) + n, as in our failed divide and conquer

attempt to speed up multiplication.

then nlogb a = nlog2 4 = n2 ;

thus f (n) = n = O(n2−ε ) for any ε < 1.

Condition of case 1 is satisfied; thus, T (n) = Θ(n2 ).

Let T (n) = 3T (n/2) + n as in the Karatsuba algorithm

then nlogb a = nlog2 3 and f (n) = n = O(nlog2 3−ε ), for any

ε < log2 3 − 1
Again, condition of case 1 is satisfied; thus, T (n) = Θ(nlog2 3 ).

COMP3121/3821/9101/9801 33 / 52
Master Theorem - Examples

Let k be a constant and T (n) = 2T (n/2) + k n, as in the Merge

Sort algorithm.

then nlogb a = nlog2 2 = n1 = n;

thus f (n) = k n = Θ(n) = Θ(nlog2 2 ).

Thus, condition of case 2 is satisfied; and so,

T (n) = Θ(nlog2 2 log n) = Θ(n log n).

COMP3121/3821/9101/9801 34 / 52
Master Theorem - Examples
Let T (n) = 3 T (n/4) + n;
then nlogb a = nlog4 3 < n0.8 ;
thus f (n) = n = Ω(n0.8+ε ) for any ε < 0.2.
Also, af (n/b) = 3f (n/4)= 3/4 n < c n = cf (n) for c = .9 < 1.

Thus, Case 3 applies, and T (n) = Θ(f (n)) = Θ(n).

Let T (n) = 2 T (n/2) + n log2 n;

then nlogb a = nlog2 2 = n1 = n.
Thus, f (n) = n log2 n = Ω(n).
However, f (n) = n log2 n 6= Ω(n1+ε ), no matter how small ε > 0.
This is because for every ε > 0, and every c > 0, no matter how
small, log2 n < c · nε for all sufficiently large n.
Homework: Prove this.
Hint: Use de L’Hôpital’s Rule to show that log n/nε → 0.

Thus, in this case the Master Theorem does not apply!

COMP3121/3821/9101/9801 35 / 52
Master Theorem - Proof: (optional material)
Since  
n
T (n) = a T + f (n) (4)
b
implies (by applying it to n/b in place of n)
     
n n n
T = aT +f (5)
b b2 b

and (by applying (1) to n/b2 in place of n)

     
n n n
T = aT +f (6)
b2 b3 b2

and so on . . ., we get
(1)
z }|
  {     
n n n
T (n) = a T +f (n) = a a T 2
+f + f (n)
b b b
| {z } | {z }
(2L) (2R)
     
2 n n n
n
+ f (n) = a2 n



=a T 2
+a f b aT +f + af b + f (n)
b b3 b2
| {z } | {z }
(3L) (3R)

3 n  
+a2 f bn2 + a f n


=a T 3 b + f (n) = . . .
b
| {z }

COMP3121/3821/9101/9801 36 / 52
Master Theorem Proof: (optional material)
Continuing in this way logb n − 1 many times we get ...
n n n
T (n) = a3 T 3 +a2 f 2 + a f + f (n) =
| {zb } b b
= ...
 n   n 
= ablogb nc T blog nc + ablogb nc−1 f blog nc−1 + . . .
b b   n  b n
b

3 n 2
+ a f 3 + a f 2 + af + f (n)
b b b
blogb nc−1
logb n
 n  X n
≈a T + ai f
blogb n i=0
bi

We now use alogb n = nlogb a :

blogb nc−1 n
X
T (n) ≈ nlogb a T (1) + ai f (7)
i=0
bi
Note that so far we did not use any assumptions on f (n) . . .
COMP3121/3821/9101/9801 37 / 52
 blogb nc−1 n
X
logb a i
T (n) ≈ n T (1) + af
i=0
bi
| {z }
Case 1: f (m) = O(mlogb a−ε )
blogb nc−1 n blogb nc−1  n logb a−ε
X X
ai f = ai O
i=0
bi i=0
bi
   
blogb nc−1 blogb nc−1 
ai
X  n logb a−ε X 
= O ai  = O nlogb a−ε 
i=0
bi i=0
(bi )logb a−ε
   
blogb nc−1  blogb nc−1 
X a i X a i
= O nlogb a−ε  = O nlogb a−ε 
blogb a−ε
i=0 i=0
blogb a b−ε
   
blogb nc−1  i blogb nc−1
X a bε  X
= O nlogb a−ε = O nlogb a−ε (bε )i 
i=0
a i=0
! m−1
ε blogb nc
(b ) −1 X qm − 1
= O nlogb a−ε ; we are using qi =
b −1
ε
i=0
q−1

COMP3121/3821/9101/9801 38 / 52
Master Theorem Proof: (optional material)
Case 1 - continued:
blogb nc−1
!
ε blogb nc
X n
logb a−ε (b ) −1
ai f i = O n
i=0
b bε − 1
  ε 
bblogb nc −1
logb a−ε
= O n 
bε − 1
nε − 1
 
= O nlogb a−ε ε
b −1
 log a
n b − nlogb a−ε

=O
bε − 1
 
= O nlogb a
blogb nc−1 n
X
Since we had: T (n) ≈ nlogb a T (1) + ai f we get:
i=0
bi
 
T (n) ≈ nlogb a T (1) + O nlogb a
 
= Θ nlogb a

COMP3121/3821/9101/9801 39 / 52
Master Theorem Proof: (optional material)
Case 2: f (m) = Θ(mlogb a )
blogb nc−1 n blogb nc−1  n logb a
X X
ai f = ai Θ
i=0
bi i=0
bi
 
blogb nc−1  n logb a
X i
= Θ a 
i=0
bi
 
blogb nc−1  i

X a
= Θ nlogb a 
i=0
(bi )logb a
 
blogb nc−1 
X a i
= Θ nlogb a 
i=0
blogb a
 
blogb nc−1
X
= Θ nlogb a 1
i=0
 
= Θ nlogb a blogb nc

COMP3121/3821/9101/9801 40 / 52
Master Theorem Proof: (optional material)
Case 2 (continued):

Thus,

blogb nc−1 n    

X
ai f = Θ nlogb a logb n = Θ nlogb a log2 n
i=0
bi

because logb n = log2 n · logb 2 = Θ(log2 n). Since we had (1):

blogb nc−1 n
X
T (n) ≈ nlogb a T (1) + ai f
i=0
bi

we get:
 
T (n) ≈ nlogb a T (1) + Θ nlogb a log2 n
 
= Θ nlogb a log2 n

COMP3121/3821/9101/9801 41 / 52
Master Theorem Proof: (optional material)
Case 3: f (m) = Ω(mlogb a+ε ) and a f (n/b) ≤ c f (n) for some 0 < c < 1.
c
We get by substitution: f (n/b) ≤ f (n)
a
c
f (n/b2 ) ≤ f (n/b)
a
c
f (n/b3 ) ≤ f (n/b2 )
a
...
c
f (n/bi ) ≤ f (n/bi−1 )
a
By chaining these inequalities we get
c c c c2
f (n/b2 ) ≤ f (n/b) ≤ · f (n) = 2 f (n)
a | {z } a |a {z } a

c c c2 c3
f (n/b3 ) ≤ f (n/b2 ) ≤ · 2 f (n) = 3 f (n)
a | {z } a |a {z } a

...
c c ci−1 ci
f (n/bi ) ≤ f (n/bi−1 ) ≤ · i−1 f (n) = i f (n)
a | {z } a |a {z } a

COMP3121/3821/9101/9801 42 / 52
Master Theorem Proof: (optional material)
Case 3 (continued):
ci
We got f (n/bi ) ≤ f (n)
ai
Thus,
blogb nc−1 blogb nc−1 ∞
X n X ci X f (n)
ai f ≤ ai f (n) < f (n) ci =
i=0
bi i=0
ai i=0
1 −c

Since we had (1):

blogb nc−1 n
X
T (n) ≈ nlogb a T (1) + ai f
i=0
bi
and since f (n) = Ω(nlogb a+ε ) we get:
T (n) < nlogb a T (1) + O (f (n)) = O (f (n))
but we also have

T (n) = aT (n/b) + f (n) > f (n)

thus,
T (n) = Θ (f (n))
COMP3121/3821/9101/9801 43 / 52
Master Theorem Proof: (optional material)

Exercise 1: Show that condition

f (n) = Ω(nlogb a+ε )

follows from the condition

a f (n/b) ≤ c f (n) for some 0 < c < 1.

Example: Let us estimate the asymptotic growth rate of T (n) which satisfies

T (n) = 2 T (n/2) + n log n

Note: we have seen that the Master Theorem does NOT apply, but the technique
used in its proof still works! So let us just unwind the recurrence and sum up the
logarithmic overheads.

COMP3121/3821/9101/9801 44 / 52
 n
T (n) = 2 T +n log n
 | {z2 } 
z  }| {
n n n
= 2 2T + log  + n log n
22 2 2
n
= 22 T 2
+n log n2 + n log n
2
| {z } 
z   }| {
2 n n n
=2 2T + 2 log 2 + n log n2 + n log n
23 2 2
n
= 23 T 3
+n log 2n2 + n log n2 + n log n
| {z2 }
...
= 2log n T 2log n
+ n log 2lognn−1 + ... + n log 2n2 + n log n


n 2
+ n log n

= nT (1) + n(log n × log n − log 2log n−1 − ... − log 22 − log 2

= nT (1) + n((log n)2 − (log n − 1) − ... − 3 − 2 − 1)

= nT (1) + n((log n)2 − log n(log n − 1)/2

= nT (1) + n((log n)2 /2 + log n/2)

= Θ n(log n)2 .

COMP3121/3821/9101/9801 45 / 52
Deterministic Linear Time Algorithm for Order Statistic

Order statistic is a generalisation of a median.

Median of a set S of n numbers is an element x ∈ S such that the
number of elements of S smaller than x and the number of
elements larger than x is either equal (if the number of elements in
S is odd) or differ by one (if the number of elements in S is even)
So if the set S was in a sorted array, the median would be in the
middle of the array.
Order statistic is the following generalisation: given a set S of n
numbers and an integer k such that 1 ≤ k ≤ n find the k th element
in size.
Clearly, this problem can be solved in time n log n by sorting S
and picking the k th element.
However, the problem can be solved in linear time by an algorithm
from 1973 by Manuel Blum, Robert W. Floyd, Vaughan Pratt,
Ron Rivest, and Robert Tarjan, which we now present.
COMP3121/3821/9101/9801 46 / 52
Deterministic Linear Time Algorithm for Order Statistic
Algorithm Select(n, i) :
Split the numbers in groups of five (the last group might contain
less than 5 elements);
Order each group by brute force in an increasing order.

>
>

>

>

>
>
>

>

>

>

>
>

>
>

>

>
>

>
>
>

>

>

>

>
>

>
>
>

>

>
>

>
>

>

>

>

>
>

>
>

>

>
>

>
>
>

>

>

>

>
>
Take the collection of all b n5 c middle elements of each group (i.e.,
the medians of each group of five).

>
>

>
>

>
>
>

>

>

>

>
>

>
>

>

>
>

>
>
>

>

>

>

>
>

>
>
>

>
>

>
>
>

>

>

>

>
>

>
>

>

>
>

>
>
>

>

>

>

>
>

Apply recursively Select algorithm to find the median p of this

collection.
COMP3121/3821/9101/9801 47 / 52
Deterministic Linear Time Algorithm for Order Statistic

Algorithm Select(n, i) continued:

partition all elements using p as a pivot;
Let k be the number of elements in the subset of all elements
smaller than the pivot p.
if i = k then return p
else if i < k then recursively Select the ith smallest element of
the set of elements smaller than the pivot.
else recursively Select the (i − k)th smallest element of the set of
elements larger than the pivot.

COMP3121/3821/9101/9801 48 / 52
Deterministic Linear Time Algorithm for Order Statistic
What have we accomplished by such a choice of the pivot?

>
>

>
>

>
>
>

>

>

>

>
>

>

>
>

>
>

>
>
>

>

>

>

>
>

>
>

>

>
>

>
>
>

>

>

>

>
>

>

>

>

>
>

>
>
>

>

>

>

>
>

>
>
>

>
>

>
>
>

>

>

>

>
>

>

>
>

>
>

>
>
>

>

>

>

>
>

>
>

>

>
>

>
>
>

>

>

>

>

>

>
>

>
>

>
>
>

>

>

>

>
>
smaller than the pivot larger than the pivot smaller or equal to the pivot larger or equal to the pivot

Note that at least b(n/5)/2c = bn/10c group medians are smaller

or equal to the pivot; and at least that many larger than the pivot.
But this implies that at least b3n/10c of the total number of
elements are smaller than the pivot, and that many elements
larger than the pivot.
(caveat: we are assuming all elements are distinct; otherwise we
have to slightly tweak the algorithm to split all elements equal to
the pivot evenly between the two groups.)
COMP3121/3821/9101/9801 49 / 52
Deterministic Linear Time Algorithm for Order Statistic
What is the run time of our algorithm?
T (n) ≤ T (n/5) + T (7n/10) + Cn.

Let us show that T (n) < 11Cn for all n. Assume that this is true
for all k < n and let us prove it is true for n as well.
Note: this is a proof using the following form of induction:
ϕ(0) & (∀n)((∀k < n)ϕ(k) → ϕ(n)) → (∀n)ϕ(n).
Thus, assume T (n/5) < 11C · n/5 and T (7n/10) < 11C · 7n/10;
then
n 7n
T (n) ≤ T (n/5) + T (7n/10) + Cn < 11C · + 11C · + Cn
5 10
Cn
= 109 < 11C · n
10

which proves out statement that T (n) < 11C · n.

COMP3121/3821/9101/9801 50 / 52
PUZZLE!

On a circular highway there are n petrol stations, unevenly spaced,

each containing a different quantity of petrol. It is known that the
total quantity of petrol on all stations is enough to go around the
highway once, and that the tank of your car can hold enough fuel to
make a trip around the highway. Prove that there always exists a
station among all of the stations on the highway, such that if you take
it as a starting point and take the fuel from that station, you can
continue to make a complete round trip around the highway, never
emptying your tank before reaching the next station to refuel.

Hint: Try proving it by induction. Find a way for reducing the case
with n + 1 petrol stations to the case with only n petrol stations by
choosing suitably a petrol station to remove, pouring its petrol to the
preceding petrol station.

COMP3121/3821/9101/9801 51 / 52
That’s All, Folks!!

COMP3121/3821/9101/9801 52 / 52