1

19
Analyzing Insertion Sort as a Analyzing Insertion Sort as a
Recursive Algorithm Recursive Algorithm
l Basic idea: divide and conquer
Divide int o 2 (or mor e) subpr oblems.
Solve each subpr oblem r ecur sively.
Combine t he r esult s.
l I nser t ion sor t is j ust a bad divide & conquer !
Subpr oblems: (a) last element
(b) all t he r est
Combine: f ind wher e t o put t he last element
Lecture 2, April 5, 2001
20
Recursion for Insertion Sort Recursion for Insertion Sort
l We get a r ecur sion f or t he r unning t ime T(n):
l For mal pr oof : by induct ion.
l Anot her way of looking: split int o n subpr oblems, mer ge
one by one.
1
2
( 1) for 1
( )
1 for 1
( ) ( 1)
( 2) ( 1)
( 3) ( 2) ( 1)
...
( )
n
i
T n n n
T n
n
T n T n n
T n n n
T n n n n
i
n

'

+ >

+
+ +
+ + +

2
21
Improving the insertion sort Improving the insertion sort
l Simple inser t ion sor t is good only f or small n.
l Balance sor t ing vs. mer ging: Mer ge equal size chunks.
l How t o mer ge:
i=1, j=1
for k=1 to 2n
if A(i)<B(j)
then
C(k)=A(i)
i++
else
C(k)=B(j)
j++
end
l O(n) t ime !!
22
Analysis Analysis
l I t er at ive appr oach:
Mer ge si ze- 1 chunks int o size- 2
chunks
Mer ge si ze- 2 chunks int o size- 4
chunks
et c.
l I nt uit ively r ight , but needs pr oof !
n
merge
n
merge
n
merge
n n
2
1
4
2
8
4 ( ) ( ) ( )
: ( log )
+ + +L
Overall
3
23
Analyzing Recursive Merge Analyzing Recursive Merge- -Sort Sort
l Anot her appr oach: r ecur sive.
Di vi de i nt o 2 equal si ze par t s.
Sor t each par t r ecur si vel y.
Mer ge.
l We dir ect ly get t he f ollowing r ecur r ence:
l How t o f or mally solve r ecur r ence ?
For exampl e, does i t mat t er t hat we have (n) i nst ead
of an exact expr essi on ??
Does i t mat t er t hat we somet i mes have n not di vi si bl e by 2 ??
2 ( / 2) ( ) 1
( )
1 1
T n n n
T n
n

'

+ >

Recursion is a way of thinking.

Easy to design recursive algorithms.
24
Summations Summations
l Bef or e dealing wit h r ecur r encies, need t o r ead Chapt er 3,
in par t icular summat ions:
2 3
1
1 1
1 1
1 1
1 1
1
1 2
1
2! 3!
1
Harmonic function: ( ) ln (1)
1 1 1
Telescoping series:
( 1) 1
1 1
1
1 1
1
1
x
n
i
n n
k k
n n
k k
n n
k k
x x
e x
H n n O
i
k k k k
k k
k k
n
=
- -
= =
- -
= =
-
= =
= + + + +
= = +

= -

+ +

= -

+

= -


= -

L
4
25
More summations More summations
l Anot her usef ul t r ick:
l Summar y:
Lear n t o r ecognize st andar d simplif icat ions
Tr y going opposit e dir ect ion
I f all f ails - apply t r icks one by one. . .
kx x
d
dx
x x
d
dx x
x
x
k
k
k
k

0 0
2
1
1 1 ( )
26
Recurrencies Recurrencies
l Chapt er 4 in t he t ext book.
l Algor it hm calls it self - r ecur sive.
l Fir st , solve f or
Claim:
Pr oof by induct ion:
1 1
( )
1
2
n
T n
n
T otherwise

'
_
1

1

1
,

+
n
k
2
T n n ( ) lg +1
T
T T
k
k k
k
k
( )
( ) ( )
lg( )
lg( )
1 1
2 2 1
2 1 1
2
2 1
1
1

+
+ +
+
+
+
+
QED
5
27
What if n not a power of 2 ? What if n not a power of 2 ?
l Easy t o pr ove by induct ion t hat
l Now we can say:
l Obser ve t hat we did not pr ove Thet a, only big- Oh !
l Technically, we should be car ef ul about f loor / ceiling, but
usually we can saf ely concent r at e on n=power of 2.
T n T n ( ) ( ) 1
lg
( ) (2 ) lg 1 (log )
n
T n T n n




= + = Q
28
Guessing the solution Guessing the solution
l I nst ead of adding sequent ially, let s divide int o 2 par t s,
add each one r ecur sively, and add t he r esult :
l Need a st r onger induct ion hypot hesis !
Assume:
Then:
( ) / 2 / 2 1 T n T n T n




= + +
Note that we omit the n=1
case for simplicity
Guess: ( ) for some constant
Then: ( ) / 2 / 2 1
/ 2 / 2 1
1 ....
T n cn c
T n T n T n
c n c n
cn Oopssss








<
= + +
< + +
= +
T n cn b c b ( ) , < for some constants
T n cn b cn b b ( ) + < > L 2 1 1 for
6
29
Another example Another example
l Consider r ecur sion:
l Fir st guess:
l We omit base case.
I nduct ion st ep:
l But we can do bet t er !
( ) 4
2
n
T n T n



= +
T n cn ( )
3
3
3 3
4 ( )
2 2 2
for 2, 1 " " 0 QED
rest
n n c
T n c n cn n n
c n rest



+ + = + -

1442443
2
2
1 2
2
2
1 2 1 2
2
1 2 2
First try: ( ) is too weak !
Assume: ( )
Then: ( ) 4 4 2
2 2 2
( )
REST
T n cn
T n c n c n
n n n
T n T n c c n c n c n n
c n c n n c n

-
= + - + = - +
= - + -
14243
30
Initial Conditions Initial Conditions
l Can init ial condit ions af f ect t he solut ion ? YES !
l n was assumed t o be a power of 2.
2
( ) ( / 2)
(1) 2 ( ) 2
(1) 3 ( ) 3
(1) 1 ( ) 1
n
n
T n T n
T T n
T T n
T T n


=
= =
= =
= =
7
31
Iterating Iterating recurrencies recurrencies
l Example:
l Disadvant ages:
Tedious
Er r or - pr one
l Use t o gener at e init ial guess, and t hen pr ove by
induct ion !
T n T n n
n n T n n n T n
n n n T n n n n T n
n n n n n T
k
i
n
n
n
n
( ) ( / )
( / ( / )) ( / )
[ / ( / )] ( / )
( )
lg
lg
lg
= +
= + + = + +
= + + + = + + +
= + + + + = +
=
-

-
-
4 2
4 2 4 4 2 16 4
2 16 4 4 8 2 4 4 8
2 4 8 2 4 1
0
1
2 1
2 1
L
1 2 4 3 4
1 2 4 3 4
Q( ) n
2 Q( ) n
2
32
Recursion Tree Recursion Tree
l Example:
l At k- t h level we get a gener al f or mula: i st eps r ight , k- i lef t
l Summing over all k, geomet r ic sum, sums t o
(over count , since T(1)=1)
T n T n T n n ( ) ( / ) ( / ) = + + 4 2
2
n
2
4
n
T


2
n
T



n
2
2
4
n


2
2
n


16
n
T


8
n
T


8
n
T


4
n
T



n
2
5
16
2
n
25
256
2
n
2
( ) ( ) 2 2
2 2
2 4 4 16
1 1 5
4 16 16
k i k i i i
i i
k k
k k
n n
i i
n n
- - - - - -









= =
= + =

Q( ) n
2
8
33
Master Method Master Method
l Consider t he f ollowing r ecur r ence:
l Mor e gener al t han t he book.
l Let . Then t he cases ar e:
Q polynomially l ar ger t han f .
f i s l ar ger t han Q by a pol yl og f act or .
Q polynomially smal l er t han f .
lg lg
lg lg
1
lg
1. ( ) ( ), 0 ( )
2. ( ) ( lg ), 0 ( lg )
( ) ( ), 0
3. ( ( ))
( / ) ( ) for some 1
b b
b b
b
a a
a a
k k
a
f n O n n
f n n n k n n
f n n
f n
af n b cf n c

+
+

= >
=
= >

<
( ) ( / ) ( ); 1, 1 T n aT n b f n a b = + >
Q n
b
a

lg
34
Build recursion tree Build recursion tree
f n ( )
f n b ( / ) f n b ( / )
f n b ( / )
2
f n b ( / )
2
f n b ( / )
2
f n b ( / )
2 ... ...
f n ( )
af n b ( / )
a f n b
2 2
( / )
...
Last row: ( elements, each one
Total:

a n
n a f n b
b b
b
b
n a
a
i i
i
n
lg lg
lg
lg
) ( ) ( ).
( ) ( / )

1
1
1
Which term dominates ?
9
35
First case: f(n) small First case: f(n) small
n
f n
n c f n cn n
a f n b ca n b cn a
b
b
cn b
b
b
n
O n
n
b
b
b
a
b
b
b
b
b
b
a
a
j j j j a j
j
j a
a j
n
a
a
lg
lg
lg
lg
lg
lg
lg
lg
( )
( ) ( ) /
( / ) ( / )
( ).
( ).
( ).)
lg
= $
= =
-
-
=
-
- -
W
Q
Q
e e
e
e
e e
e
e
e
e
s.t for "large enough n",
The ratio summed over all possible j:
Total:
Lower bound is trivial (Why ?? First term in the original expression was already
1
1
36
Second case Second case
lg
lg
lg
lg
1
lg
( ) ( lg )
lg (lg ) (there are (lg ) elements in the sum)
This is an UPPER bound ! How to prove the lower bound ??
Rough and easy ap
b
b
b
a k
b
a
k
a
a j k k
j j
n
n
f n n n
n n
a O n n O n
b b
+

=



= Q
=

144244314243
lg 1 (lg )/2 (lg )/2
1
1 1 1
proach:
lg lg lg (const) lg
(Note that we use the assumption that 0)
b b b
n n n
k k k k
j j
i i i
n n
n n
b b
k
-
+
= = =



=

10
37
Third case Third case
a f n b c f n c f n n
c f n f n
a f n b O f n
n O f n
f n
j j j a
j
i
n
j j
i
n
a
b
b
b
b
( / ) ( ) , ( ) ( )
( ) ( ( ))
( / ) ( ( ))
( ) ( ( ))
( ( ))
lg
lg
lg
lg
<

for some and

Note Big- Oh and not Theta !
The first term is already
TOTAL:
1
1
1
1
1