1 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Chapter 1: Introduction
1.0 What is an Algorithm?
Before we start talking about algorithms for an entire semester, we should try to figure out what
an algorithm actually is. We could, for example, have a short look at how others define an
algorithm:
Definition 1:
An algorithm is a computable set of steps to achieve a desired result.
(found on numerous websites)
Definition 2:
An algorithm is a set of rules that specify the order and kind of arithmetic operations that
are used on specified set of data.
Definition 3:
An algorithm is a sequence of finite number of steps arranged in a specific logical order
which, when executed, will produce a correct solution for a specific problem.
(taken from an undergraduate course at UMBC).
Definition 4:
An algorithm is a set of instructions for solving a problem. When the instructions are
followed, it must eventually stop with an answer.
(given by Prof. Carol Wolf in a course on algorithms).
Definition 5:
An algorithm is a finite, definite, effective procedure, with some output.
(given by Donald Knuth)
27.11.2003 17:18
Fundamental Algorithms
2 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
27.11.2003 17:18
Fundamental Algorithms
3 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Question:
How many arithmetic operations does it take to compute f j using the algorithm Fibo?
That means, we neglect the costs of all operations except the arithmetic ones. Thus, we will
basically count the number of additions.
Define:
T Fibo(n) = number of arithmetic operations (+,,*,/) that Fibo will perform with n as input
parameter.
Examining the function Fibo ,we see that:
T Fibo(0) = T Fibo(1) = 0, as there are no additions performed, if the parameter n is 0 or 1.
T Fibo(n) = T Fibo(n 1) + T Fibo(n 2), if the parameter n is larger than 1.
In that case, the number of additions is the sum of the additions performed by the two
recursive calls.
Such a recursive characterization of the number of operations is very often found for recursive
algorithms. We will soon examine techniques to solve such recurrence equations. In the
meantime, we try to stick to more basic techniques.
0
1
1
1
2
2
3
3
4
5
5
8
6
13
7
21
8
34
T Fibon
12
21
36
60
99
Proposition:
After a (very) close look at the values given in this table, we propose that
T Fibo(n) = 3 f n 3
Proof:
27.11.2003 17:18
Fundamental Algorithms
4 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
3 + (3 f
3( f
3 fn 3
n1
n1
3) + (3 f
n2
n2
3)
)3
(q.e.d.!)
2 2 fn 2 n
to get a more precise estimate of Fibo's operation count.
We will prove the two inequalities separately, by induction. In either case, we use that f j+1 f j
for all j 0 (proof left to the reader....)
Proof for f n 2 n:
By induction over n:
case n = 0 : then f 0 = 1, which is equal to 2 0 = 1
case n = 1: then f 1 = 1, which is smaller than 2 1 = 2
case n 2 (induction step):
f n = f n1 + f n2 f n1 + f n1 = 2 f n1 2 2 n1 = 2 n
n
Proof for f n 2 2 :
case 1: n = 2k n2 = k
again, we prove this by induction (over k):
for k = 0, which means that n = 2 k = 0, we have f 0 = 1, which is equal to 2 0 = 1.
for k 1 (i.e. n 2), we compute that f 2k = f 2k1 + f 2k2 2 f 2k2 = 2 f 2(k1).
By induction assumption f 2(k1) 2 k1, and therefore f 2k 2 2 k1 = 2 k .
case 2: n = 2 k + 1 n2 = k
27.11.2003 17:18
Fundamental Algorithms
5 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
3 2 2 3 T Fibo(n) 3 2 n 3
Hence, the operation count of Fibo increases exponentially with the size of the input parameter.
Example
How long would a typical computer take to compute the 100th fibonacci number using
the algorithm Fibo?
Answer: n = 100 T Fibo(n) 3 2 50 3 10 15, i.e. the computer has to perform more than 10 15
additions.
If we assume that our computer is able to perform one arithmetic operation per nanosecond
(compare this to the typical GHz clock rate of current processors), the execution time would
be at least 39 days!
Exercise:
How large is the respective upper bound for the computing time?
Remark:
In our estimate of Fibo's operation count, we have used a rather weak lower bound of the size of
the fibonacci numbers. A well known algebraic formulation of the fibonacci numbers claims that
n
n
5 1
1 5 + 1
2
.
f n = 2
5
This leads to a quite accurate estimate of Fibo's operation count: T Fibo(100) 10 21.
This means, that under the conditions above (1 operation per nanosecond), the computation of
the 100th fibonacci number will take approximately 31 500 years, which by far exceeds our initial
estimate of "at least 39 days" . . .
27.11.2003 17:18
Fundamental Algorithms
6 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
We can easily see that intermediate results like Fibo(2) or Fibo(1) are computed again (and again
and again..). Hence, we should try to store these intermediate results, and reuse them.
Strategy
Use two variables, last2, and last1, to store the last two fibonacci numbers f n 2 and
f n 1, respectively.
Use one additional variable, f, to compute f n.
Iterative Algorithm
Fibit (x : Integer) : Integer {
if x < 1 then return 1;
else {
last2 := 1;
last1 := 1:
for i from 2 to x do {
f := last2 + last1;
last2 := last1;
last1 := f;
}
}
}
Question:
How many arithmetic operations does it take to compute the nth Fibonacci number
using algorithm Fibit?
Answer
For n 1: 0 operations (return 1 as the result)
for n 2: There is 1 operation per cycle of the forloop.
The loop is executed n 1 times, thus we need n 1 operations.
Therefore,
27.11.2003 17:18
Fundamental Algorithms
7 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
for n 1
0
T Fibit(n) =
n 1 for n 2
We can see that the operation count of Fibit increases linearly with the size of the input
parameter.
Example
Again, we imagine a computer that performs 1 arithmetic operation per nanosecond. The number
of arithmetic operations to compute the 100th fibonacci number using algorithm Fibit is now T Fibit
(100) 99. Hence, the computing time will be only 99 nanoseconds!
Remark
Please, do not take this example as one to show that recursive programming, on itself, is slow.
It's not recursion that makes Fibo slow, it's excessive recomputation of intermediate results.
27.11.2003 17:18
Fundamental Algorithms
8 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Execution of a RAMprogram:
To "run" a program in the RAM, we therefore need to:
define the program, i.e. the exact list of statements.
define starting values for the registers (the input).
define starting values for the program counter (usually, we'll start with the first statement).
Statements of a RAM
Notation:
<Ri>
<Ri> := x
=
=
List of Statements:
Statement
Ri Rj
Effect on registers
<Ri> : = <Rj>
Program Counter
<PC> : = <PC> + 1
Ri RRj
RRi Rj
Ri k
Ri Rj + Rk
<Ri> : =
<R<Ri>>
<Ri> : =
<Ri> : =
<PC>
<PC>
<PC>
<PC>
<R<Rj>>
: = <Rj>
k
<Rj> + <Rk>
:
:
:
:
=
=
=
=
<PC>
<PC>
<PC>
<PC>
+
+
+
+
1
1
1
1
27.11.2003 17:18
Fundamental Algorithms
9 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Ri Rj  Rk
<Ri> : =
max {0, <Rj>  <Rk>}
<PC> : = <PC> + 1
GOTO m
IF Ri=0 GOTO
m
<PC> : = m
<PC> : =
if <Ri> = 0
m
<PC> + 1 otherwise.
IF Ri>0 GOTO
m
<PC> : =
if <Ri> > 0
m
<PC> + 1 otherwise.
Using the well known identity x y = x, we will attempt to multiply x by y by adding up x exactly y
i=1
times.
A respective algorithm in our previous notation could, for example, look like this:
R3
IF
R2
R1
R1
1
= 0 GOTO 5
R2 + R0
R1  R3
27.11.2003 17:18
Fundamental Algorithms
10 of 67
(4)
(5)
(6)
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
GOTO 1
R0 R2
STOP
Example:
Let's examine the working steps of our RAM on the special input x=4, and y=3:
< PC >
< R0 >
< R1 >
< R2 >
< R3 >
after step
==========================================================
0 (at start)
0
4
3
0
0
1
1
4
3
0
1
2
2
4
3
0
1
3
3
4
3
4
1
4
4
4
2
4
1
5
1
4
2
4
1
6
2
4
2
4
1
7
3
4
2
8
1
8
4
4
1
8
1
9
1
4
1
8
1
10
2
4
1
8
1
11
3
4
1
12
1
12
4
4
0
12
1
13
1
4
0
12
1
14
5
4
0
12
1
15
6
12
0
12
1
16
stop
We can see that our RAM performs three (for y=3) iterations of its basic loop (statements 1 to 4).
Each of the loop iterations requires 4 work cycles of the RAM. The remaining statements require
3 work steps (one before, and two after the loop iterations). Hence, for y=3, our RAM performs 15
work steps. For general y, it is easy to see that the number of work cycles is 1 + 4 y + 2 = 3 + 4 y.
Note, that the number of work cycles is independent of x!
27.11.2003 17:18
Fundamental Algorithms
11 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Example
uni
The Multiplication RAM has a uniform time complexity of T M
(x, y) = 3 + 4 y, independent of x. The
uniform size of its input is x uni = 2, independent of the size of x, and y.
Exercise
Discuss, whether the uniform complexity of a RAM is a good model for measuring complexity of
real algorithms in real computers.
x log = def l (x i) ,
i=0
Remarks:
for x : l(x) = log 2 (x) + 1 = log 2 ( x + 1)
idea for proof: with k binary digits 2 k different integers can be represented, which implies
that, if l(x) = k, then x is between 2k1, and 2k.
if we use the decimal system (instead of the binary system), its log 10 instead of log2 (similar
for all other systems ...). We will always use the binary system, and write log instead of
log2.
Logarithmic Costs
l( < Rj > ) + 1
Ri RRj
RRi Rj
Ri k
Ri Rj + Rk
Ri Rj  Rk
27.11.2003 17:18
Fundamental Algorithms
12 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
GOTO m
IF Ri=0 GOTO m
IF Ri>0 GOTO m
1
l( < Ri > ) + 1
log
Example:
We compute the logarithmic costs for an execution of the MultiplicationRAM on input (x,y). We
can compute the logarithmic costs by summing up the costs for each loop cycle separately:
for statement (0):
ly + 1
l0 + lx + 1
ly + l1 + 1
ly 1 + 1
lx + lx + 1
ly 1 + l1 + 1 +1
l i x + lx + 1
ly i + l1 + 1
+1
l (y 1)x + lx + 1
l1 + l1 + 1
+1
...
+1
...
+
ly i + 1
...
...
l1 + 1
l0 + 1
ly x + 1
T M (x, y)
y1
y1
y1
i=0
i=0
i=0
l y i + ylx + l i x + l y i + (1 + 1 + l1 + 1 + 1)y + 4 + l0 + l x y
y
y1
i=1
i=0
5 + (4 + l1 + lx)y + 2 li + l i x + l x y
y1
i=1
i=0
log
As the logarithmic size of the input, (x, y) log = lx + ly, we may conclude that T M (x, y) 5 + y
(5 + 3 (x, y) log) + (x, y) log
Hence, we can get a relation between the size of the input, and the logarithmic costs.
27.11.2003 17:18
Fundamental Algorithms
13 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
i.e. for all inputs of uniform size n, the uniform time complexity has to be bounded
by tn.
log
Remarks:
uni
uni
TM
n = sup{T M
n : x uni = n}
log
log
T M n = sup{T M n : x log = n}
Example: (MultiplicationRAM)
uni
The Multiplication RAM's uniform time complexity, T M
(x, y) = 4 y + 3, is independent of the
uniform size of its input, (x,y) uni = 2, which is independent of x, and y. There is no function tn
uni
x,y = 4 y + 3 t2, because y can become arbitrarily large.
such that the size of the input T M
Thus, M is not uniformly tntimebounded for any function t.
log
which means exactly that M is logarithmically time bounded w.r.t. the function tn = 5 + 2 n (5 + 3 n)
+n. We can therefore say that
"M has an exponential timecomplexity w.r.t. the logarithmic complexity
measure".
Exercise
Discuss, in which situations the logarithmic time complexity can be a better model for the
characterisation of computing time than the uniform time complexity. Consider the following
examples:
sorting a large set of numbers (phone numbers, for example), the size of the numbers is
within a fixed range;
computing the prime factorization of a single, very large integer
computing a matrixvector product, or solving a large linear system of equations
Discuss how the word length (i.e. the number of bits a CPU can process in a single step) affects
whether uniform or logarithmic complexity is more appropriate.
Fundamental Algorithms
14 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Definition:
Given functions f,g: + , then we define
1. fO(g) def c>0 n 0 n n 0 :f(n)cg(n)
2. f(g) def c>0 n 0 n n 0 :f(n)cg(n)
3. f(g) def fO(g) and f(g)
4. fo(g) def c>0 n 0 n n 0 :f(n)cg(n)
5. f(g) def c>0 n 0 n n 0 :f(n)cg(n)
Remarks:
1. if fO(g) , then g is called an asymptotic upper bound of f;
2. if f(g) , then g is called an asymptotic lower bound of f;
3. if f(g) , then g is called an asymptotically tight bound of f;
4. if fo(g) , then f is called an asymptotically smaller than g;
5. if f(g) , then f is called an asymptotically larger than g;
Further Remarks:
1. in the definition for fo(g) , we can replace the condition f(n)cg(n) by the equivalent
condition f(n) g(n) c , thus the definition can also be written as c>0 n 0 n n 0 : f(n)
g(n) 1 c , which is equivalent to lim n f(n) g(n) =0 .
2. in the same way, f(g) is equivalent to lim n g(n) f(n) =0
3. from these two observation, we may conclude that fo(g)g(f)
In literature, you will also often find notations like f=O(g) , instead of fO(g) .
Remark:
The definitions for the asymptotical bounds may be used for multidimensional functions f,g: k
+ , too, if we replace the terms n> n 0 by n=( n 1 ,..., n k ): n i n 0 .
27.11.2003 17:18
Fundamental Algorithms
15 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Comparison of functions
The O,,,o, and notations define relations between functions. We can therefore ask
whether these relations are transitive, reflexive, symmetric, etc.:
transitive: all of the five relations are transitive!
reflexive: only O,, and are reflexive
symmetric: f(g)if and only ifg(f)
transpose symmetry:
fO(g) if and only if g(f)
f(g) if and only if gO(f)
2. Sorting
Definition of the Sorting problem:
Input:
A sequence of n numbers a 1, a 2, a 3, . . . ,a n
Output:

, . . . , a n.
Data Structure:
array A[1..n] containing the sequence a 1 (in A[1]), . . ., a n (in A[n]).
27.11.2003 17:18
Fundamental Algorithms
16 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Algorithm:
INSERTSORT(A:Array[1..n]) {
for j from 2 to n {
// insert A[j] into sequence A[1..j1]
key := A[j];
i := j1;
Loop invariant:
Before (and after) each iteration of the forloop,the subarray A[1..j1] consists of all
elements originally in A[1..j1], but in sorted order.
To prove that an invariant is true, we always have to show its correctness at the beginning
(initialization), that it stays true from one iteration to the next (maintenance), and show of what
use it is for the correctness at termination of the algorithm:
Initialization (invariant is true before the first iteration of the loop):
The loop starts at j = 2, therefore A[1..j1] consists of the single element A[1].
A subarry containing only one element is, of course, sorted. The loop invariant is therefore
correct.
Maitenance (if the invariant is true before an iteration, it remains true before the next iteration):
The while loop will shift all elements that are larger than the additional element A[j] to the
right (by one position). A[j] will be inserted at the empty (and correct) position. Thus,
A[1..j] is a sorted array. A formal proof (especially that the position was the correct one)
would f.e. state (and prove) a loop invariant for the whileloop.
Termination (on termination, the loop invariant helps to show that the algorithm is correct)
The forloop terminates when j exceeds n (that means j = n + 1 ). Thus, at termination, A[1
.. (n+1)1] = A[1..n] will be sorted, and contain all original elements. Hence, the
algorithm INSERTSORT is correct.
27.11.2003 17:18
Fundamental Algorithms
17 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
cost
repetitions
c1
c2
n1
c3
tj
for j from 2 to n do {
key := A[j]; i := j1;
while i>=1 and A[i]>key {
j=2
n
(t j 1)
c4
j=2
n1
c5
}
}
where t j denotes the number of times the whileloop is executed in the jth forloop.
Thus, for the total costs T (n) of the algorithm, we get
n
j=2
j=2
T (n ) = c 1 n + (c 2 + c 5 )(n 1 ) + c 3 t j + c 4 (t j 1 )
The total costs of INSERTSORT naturally depend on the data, which determine how often the
while loop will be repeated in each for loop.
27.11.2003 17:18
Fundamental Algorithms
18 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
T (n )
j=2
j=2
c 1 n + (c 2 + c 5 )(n 1) + c 3 j + c 4 ( j 1 )
n (n + 1)
n (n 1)
1 + c 4
c 1 n + (c 2 + c 5)(n 1) + c 3
2
2
1
c3 c4
(c 3 + c 4) n 2 + c 1 + c 2 + + c 5 n (c 2 + c 3 + c 5)
2 2
2
T (n ) =
x X (n)
P(x) T (x)
On the average, the number t j of repetitions of the while loop will be 2 , because  on the average
 about half of the elements in A[1..n1] can be expected to be larger than A[j].
_
T (n )
n
j
j
+ c 4 1
j=2 2
j = 2 2
n
c 1 n + (c 2 + c 5 )(n 1) + c 3
c 1 n + (c 2 + c 5)(n 1) +
1
c3 c4
c3
(c 3 + c 4) n 2 + c 1 + c 2 + + c 5 n c 2 + + c 5
4 4
2
4
1 n (n + 1)
1 n (n 1)
1 + c 4
c3
2 2
2
2
27.11.2003 17:18
Fundamental Algorithms
19 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Basic ideas
compare neighbouring elements only
exchange values if they are not in sorted order
repeat until array is sorted
Algorithm
Algorithm
BUBBLESORT(A:Array[1..n]) {
cost
for i from 1 to n do {
for j from n downto i+1 do {
repetitions
c1
n+1
c2
(n i + 1 )
i=1
( n i)
c3
i=1
n
( n i)
c 4 t i, j
i=1
}
}
}
0 , if A[j] and A[j1] are in correct order in the i th loop
where t i, j =
1 otherwise
Costs:
Best case: all t ij = 0
T (n )
i=1
i=1
c 1 ( n + 1 ) + c 2 ( n i + 1 ) + c 3 ( n i)
c 1(n + 1) + c 2 k + c 3 k
c 1(n + 1) +
n1
k =1
k =1
c2
c3
n(n + 1) + n(n 1) (n 2)
2
2
27.11.2003 17:18
Fundamental Algorithms
20 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Correctness of BUBBLESORT:
To give a proof for the correctness of BUBBLESORT, we can for example use the following two
loop invariants:
after each cycle of the iloop:
A[1..i] will contain the i smallest elements in sorted order.
after each cycle of the jloop:
the smallest element of A[(j1)..n] will be in A[j1].
2.3. Mergesort
Basic idea: "divideandconquer"
Divide the problem into two (or more) subproblems:
split the array into two arrays of equal size
Conquer the subproblems by solving them recursively:
sort both arrays using MERGESORT.
Combine the solution of the subproblems:
merge the two sorted array to produce the entire sorted array.
27.11.2003 17:18
Fundamental Algorithms
21 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
MERGE(L,R,A);
Thus, we have to find a function that satifies this socalled recurrence equation.
27.11.2003 17:18
Fundamental Algorithms
22 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
T MS(n) = T MS(2 k)
c 2 2 k + 2 T MS(2 k 1)
c 2 2 k + 2 c 2 2 k 1 + 2 2 T MS(2 k 2)
...
c 2 2 k + 2 c 2 2 k 1 + . . . + 2 i 1 c 2 2 k i + 1 + 2 i T MS(2 k i)
...
c 2 2 k + 2 c 2 2 k 1 + . . . + 2 k 1 c 2 2 1 + 2 k T MS(1)
k c2 2k + 2k c1
c 2 n log 2 n + c 1 n
T MS(n) (n log n)
c 2 n log 2 n + c 1 n
c 2 n log 2(2 n2 ) + 2 c 1 n2
c 2 n(1 + log 2 n2 ) + 2 c 1 n2
c 2 n + c 2 n log 2 n2 + 2 c 1 n2
2.4. Quicksort
Divideandconquer algorithm:
Divide:
the elements of the input array A[p..r] are rearranged such that the subarray A[p..q]
contains no element that is larger then any element of A[q+1 .. r]. The respective
index q is computed by the partitioning procedure.
Conquer:
the subarray A[p..q] and A[q+1 .. r] are both sorted using quicksort
Combine:
both subarrays, A[p..q] and A[q+1 .. r], will then be sorted, and all elements of
A[p..q] will be less than or equal to the minimal element of A[q+1 ..r].
Therefore, the entire array A[p..r] is sorted, and no further work is necessary.
27.11.2003 17:18
Fundamental Algorithms
23 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Remark:
Like INSERTSORT, and BUBBLESORT, quicksort sorts "in place".
}
return j;
2.4.2. Quicksort
27.11.2003 17:18
Fundamental Algorithms
24 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
BestCase:
The best case occurs when each partitioning splits the array into two partitions of equal size.
Then:
n
T (n) = 2 T + (n) T (n) (n log n)
2
See the analysis of the time complexity of MERGESORT for the solution of the respective
recurrence.
WorstCase:
The worst case occurs, when each partitioning will lead to one partition that contains only one
element. Then:
T (n) = T (n 1) + T (1) + (n) = T (n 1) + (n)
Applying this successively, we get:
T (n )
T (n 1 ) + (n )
T (n 2 ) + (n 1 ) + (n )
...
T (1) + (2 ) + . . . + (n 1) + (n )
(k) = k = (n 2)
k = 1
k =1
Thus, in the best case, quicksort is (asymptotically) as fast as mergesort, but in the worst case, it
is as slow as insertsort, or bubblesort!
Remark
27.11.2003 17:18
Fundamental Algorithms
25 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
The worst case occurs, if A is already in sorted order, or sorted in reverse order. QUICKSORT will
also be especially slow, if the array is nearly sorted, because several of the partitions will then be
sorted arrays, and suffer from worst case time complexity.
return q;
if p < r then {
q := RAND_PARTITION(A);
RAND_QUICKSORT (A[p...q]);
RAND_QUICKSORT (A[q+1 ..r]);
}
1
n
27.11.2003 17:18
Fundamental Algorithms
26 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Basic idea:
Assume that A contains k numbers that are smaller than the pivot element. Then:
for 0 < k < n, PARTITION will create partitions of size k and n k.
for k = 0, PARTITION will create partitions of size 1 and n 1.
the probability of k = 0, k = 1, . . ., k = n 1, is always 1n .
T (n ) =
all cases i
P(case i) T (case i)
T (n) (n log n)
27.11.2003 17:18
Fundamental Algorithms
27 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Medianpartitioning
Choosing the median element of the array A would, of course, lead to a perfectly balanced
partitioning in each step, and therefore, to a guaranteed (n log n) time complexity.
However, this requires a fast algorithm to find the median of an array (see chapter 3).
Exercise:
Given the recurrence
T (n ) =
2 n1
T (k) + c n ,
n k=1
show that
T (1) = 1
Proof by induction:
Claim:
Case n = 1:
T (1) = 1, and 4(c + 1) 1 log 1 + 1 = 4(c + 1) 0 + 1 = 1.
Induction step: n 1 n:
using the induction assumption, T (k) 4(c + 1) k log k + 1 for all k = 1, . . . ,n 1, we may
conclude that:
T (n )
2 n 1
T (k) + c n
n k =1
2 n 1
(4 (c + 1) k log k + 1) + c n
n k =1
8 (c + 1) n 1
(k log k ) + 2 + c n
n
k =1
2
nk = 11 (k log k ) 12 n 2 log n n8
T (n )
to get
8 (c + 1) 1 2
n2
+ 2 + cn
n log n
8
n
2
4 n(c + 1) log n (c + 1) n + 2 + c n
4(c + 1) n log n + 2 n
4(c + 1) n log n + 1
Thus, our proof by induction is complete. What remains to be done, is to prove the lemma
n 1
(k log k )
k =1
1 2
n2
n log n
8
2
27.11.2003 17:18
Fundamental Algorithms
28 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Proof of lemma:
We set m := n2 , and split the sum into two halves:
n1
(k log k )
k =1
m 1
n1
k =1
k =m
(k log k ) + (k log k )
n1
m1
n 1
n m1
log k + log n k = (log n 1) k + log n k
2 k = 1
k=m
k =1
k =m
log n k k
1
1
log n n(n 1) m(m 1)
2
2
1 2
1 n n
n log n 1
2
2 22
1 2
1
n log n n 2
2
8
n1
m 1
k =1
k =1
2.5. Heapsort
2.5.1. Heaps
A (binary) heap is a data structure based on a binary tree, but is stored in a array. In any
node of the tree, the stored value shall be greater than that in its two child nodes.
Example:
Array representation:
15
13
11
27.11.2003 17:18
Fundamental Algorithms
29 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Remarks:
the binary tree has to be completely filled, i.e."missing nodes" may only occur on the lowest
level, and not left of any other node on the lowest level;
the numbering of the nodes of the tree is called "breadthfirst numbering".
Observation:
There are three simple functions to compute the index of the parent, the left son, and the right
son of a node i:
27.11.2003 17:18
Fundamental Algorithms
30 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
HEAPIFY first determines which of the two children of node i is the larger one. If this child is
larger than i, it exchanges it with its parent. As this might violate the heap property in the
respected child node, HEAPIFY is recursively called for that heap.
BUILDHEAP (A:Array[1..n]) {
heapsize := n;
for i from n/2 downto 1 do {
HEAPIFY(A, heapsize, i);
}
}
We skip all elements of the lowest level, because they simply don't have an children that might
not satisy the heap property.
T (n ) = (h i)
i=1
Fundamental Algorithms
31 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
T (n)
h
h (k)
h
n
k
k + 1 (k) n k + 1 = n k +1
k = 1 2
k =1 2
k =1 2
1
n
n h k
k 2 2 (n )
1
2 k = 1 2
2 (1 2 )
k
2k
x
.
(1 x) 2
HEAPSORT (A:Array[1..n]) {
BUILDHEAP(A);
for heapsize from n downto 2 do {
exchange A[1] and A[heapsize];
HEAPIFY(A, heapsize1, 1);
}
}
Each iteration of the for loop will determine put the largest of the elements in A[1..heapsize]
in A[heapsize] (after the exchange with A[1]). As the heapsize will then be reduced, the
respective element will no longer be affected by the algorithm, and stay in its correct(!) position.
27.11.2003 17:18
Fundamental Algorithms
32 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Comparisons sorts are sorting algorithms that use only comparisions (i.e. tests like ,
=, <, . . .) to determine the relative order of the elements.
Examples:
Mergesort, Quicksort, Heapsort, Insertsort, and Bubblesort are all comparison sorts.
Example:
The following picture shows a decision tree for sorting 3 elements a 1 , a 2 , a 3
Observations:
Each comparison sort can be represented by a decision tree.
A decision tree can be used as a comparison sort, if every possible permutation is
annotated to at least one leaf of the tree.
to sort a sequence of n elements, a decision tree needs to have at least n ! leaves.
Theorem:
27.11.2003 17:18
Fundamental Algorithms
33 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Any decision tree that sorts n elements has height (n log n).
Proof:
To sort n elements, a decision tree needs n ! leaves, and a tree of height h has at most 2 h leaves.
using h comparisons in the worst case, we can sort n elements only if n ! 2 h, or h log(n !).
n
Using the wellknown inequality n ! n 2 (see exercise (b) on worksheet 2), we get
n
n
h log(n !) logn 2 = log(n)
2
Corollary:
Mergesort and heapsort are asymptotically optimal comparision sorts.
That means no comparison sort can be asymptotically faster than mergesort or heapsort. Thus, if
we want to find faster algorithms, we have to look for ways to determine the correct order of
elements that are not based on comparing the elements.
27.11.2003 17:18
Fundamental Algorithms
34 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
for i from 2 to k do {
C[i] := C[i1] + C[i]
};
// C[i] now contains the number of elements i
for j from n downto 1 do {
pos := C[A[j]];
}
}
B[pos] := A[j];
C[A[j]] := C[A[j]]  1 ;
return B;
Time compexity of counting sort is (k + n), as we can easily see from the number of iterations in
each for loop.
Thus, if k O(n), then counting sort only needs (n) operations!
Time complexity:
The time compexity of RADIXSORT is (d (n + k )).
If the decimal system is used, then k=10, and the time complexity is (d n). Naturally the (d n)
complexity also holds for the use of the binary (octal, hexadecimal, ...) system.
If, in addition, the size of the input numbers is bounded, then d is bounded, and the time
complexity is (n).
27.11.2003 17:18
Fundamental Algorithms
35 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
BUCKETSORT (A:Array[1..n]) {
Create Array B[0..n1] of Buckets;
//assume all Buckets B[i] are empty at first
for i from 1 to n do {
insert A[i] into Bucket B[i];
}
for i from 0 to n1 do {
sort Bucket B[n A[i]];
}
}
We assume that we can concatenate the buckets in O(n) time, which should not be difficult to
achieve: we can, for example, simply copy all n elements into an output array.
27.11.2003 17:18
Fundamental Algorithms
36 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
n 1
1
P(n i = k ) = 1
k n
k
nk
From probability theory, we know (or we can look it up) the expected value and the empirical
variance:
E[n i] = n
1
1
1
1
= 1 and Var[n i] = n 1 = 1
n
n
n
n
Now, the expected time to sort all buckets, for example by using INSERTSORT, is:
n1
n 1
O(E[n 2i ]) = O E[n 2i ]
i=0
i = 0
1
= ( 1 )
n
n 1
n 1
i = 0
T (n) =
n
2 T 2 + c 2 n for n > 1
Remark:
Oftentimes, a recurrence will also be given using the  ( O , , . . . )notation:
for n 1
(1)
T (n) =
n
2 T 2 + (n) for n > 1
27.11.2003 17:18
Fundamental Algorithms
37 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Example
We try to solve the recurrence above:
for n 1
c 1
T (n) =
n
2 T 2 + c 2 n for n > 1
step 1:
we guess the type of the solution: T (n) = a n log 2 n + b n
step 2:
we determine the correct values for the parameters a, and b, and prove that the resulting
function satisfies the recurrence:
for n = 1: T (1) = a 1 log 2 1 + b 1 = b; the recurrence is therefore satisfied if T (1) = c 1
=b
for n > 1, we insert our proposed solution into the recurrence equation:
n
n
n
a n log 2 n + b n = 2a log 2 + b + c 2 n
2
2
2
a n log 2 n + b n
a n(log 2 n 1) + b n + c 2 n
a n + c 2 n
a = c2
Therefore, the solution for the recurrence is T (n) = c 2 n log 2 n + c 1 n.
27.11.2003 17:18
Fundamental Algorithms
38 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Example:
See our analysis of the time complexity of MERGESORT in section 2.3.2 for an example of this
approach.
(1 )
T (n )
n
a T + f (n)
b
Remarks:
If in the term a T ( nb ) the fraction
n
b
Proof:
The proof of the master theorem is left as an exercise for the reader 1. Some technicalities of the
proof, however, need some short comments:
in case 1, f (n) O(n log b a) is not sufficient.
in case 3, f (n) (n log b a) is not sufficient.
Instead, f (n) has to be polynomially larger (or smaller) than n log b a, that means f (n) O
(n log b a n ), or f (n) (n log b a n ), respectively, where has to be a positive (nonzero)
constant. Otherwise, the master theorem is not applicable!
27.11.2003 17:18
Fundamental Algorithms
39 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
aT
=a
= a log a = a
= n log b a = T (n)
b
b
b
a
b
The master theorem compares the nonrecursive part of the costs, f (n), with this solution (the
recursive part):
in case 1, the costs of the recursion dominate, such that T (n) (n log b a)
in case 2, the costs of recursion and that of the combination are about the same, therefore
T (n) (n log b a log n)
in case 3, the costs for the nonrecursive combination dominate: T (n) ( f (n))
Examples:
Mergesort:
n
T (n) = 2 T + f (n) , where f (n) (n)
2
To apply the master theorem, we set a = 2 , and b = 2, which means that n log b a = n log 2 2 = n.
As f (n) (n), case 2 of the master theorem applies, and therefore T (n) (n log n), as
expected.
Other recurrence:
2n
T (n) = 2 T + f (n) , where f (n) (n)
3
a = 2 , and b = 32 , and consequently n log b a = n
log 3 2
2
n 1.709....
f (n) (n) implies f (n) O(n 1.7) such that case 1 of the master theorem applies.
log 3 2
Therefore T (n) (n 1.709...) = n 2 .
just kidding . . . the proof of the master theorem is beyond the scope of this course. We
therefore refer to Cormen et. al.
27.11.2003 17:18
Fundamental Algorithms
40 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
3. Selecting
Definition: (selection problem)
Input:
a set A of n (distinct) numbers, and a number i , with 1 i n.
Output:
the element y A that is larger than exactly i 1 other elements of A.
Question
Is there a faster (i.e (n)) algorithm?
MINIMUM(A:Array[1..n]) : Integer {
min := A[1];
for i from 2 to n do {
if min > A[1] then min := A[i];
}
return min;
}
MAXIMUM(A:Array[1..n]) : Integer {
max := A[1];
for i from 2 to n do {
if max < A[1] then max := A[i];
}
return max;
}
It is also reasonably easy to imagine an O(n)algorithm that finds the second (third, fourth, . . .)
smallest/largest element of an array: in contrast to the algorithms MINIMUM, and MAXIMUM we
would have to store the two (three, four, . . .) currently smallest/largest elements in respective
27.11.2003 17:18
Fundamental Algorithms
41 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
variables min1, min2 (min3, min4, . . .), and update them accordingly while we loop through
the array A.
Hence, the crucial question is, whether we can transfer this O(n)performance to an algorithm that
finds the ith largest/smallest element, where i is given as a parameter.
3.2 Quickselect
Quickselect adopts the partitioning idea of quicksort for the selection problem. We will use the
algorithmn RAND_PARTITION for the partitioning.
if i k
then // the ith largest element is in the first partition
return RANDSELECT(A[p..q], i)
else // the ith largest element is in the second partition
// (take into account the k elements in the first partit
return RANDSELECT(A[q+1 .. r], ik);
2
n
1
n
The cost of randselect is O(n) for the partitioning plus the cost for one recursive call (instead of
two for quicksort). The cost of the recursive call is determined by the size of the subarray. We
want to compute the average worstcase complexity, so we
assume the worst case in the sense that we assume that we always have to make the
27.11.2003 17:18
Fundamental Algorithms
42 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
T (n )
n 1 _
1 _
n1 _
1 _
T (n 1) + 2 T (k) + O(n)
n
k = n
_
2 n1 _
T (k) + O(n) , because T (n 1) O(n 2)
n k = n
2
T (n )
2 n1 _
T (k) + O(n)
n k = n
2
by substition:
assume that the O(n)term is d n.
_
T (n )
2 n1 _
T (k) + O(n)
n k = n
2 n1
ck + d n
n k = n
2
n
21
2 c n 1
k k + d n
n k = 1
k =1
n
n
2 c n (n 1) 2 ( 2 1)
+dn
n 2
2
c n
n
c(n 1) 1 + d n
2
n2
3
1
c n + d n
2
4
c
c
c n + d n n
4
2
_
Now, we pick c large enough such that d n 4c n 2c < 0, which is equivalent to 4c n + 2c > d n. Then T
27.11.2003 17:18
Fundamental Algorithms
43 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
x := BFPRT_SELECT(M, cols
2 );
k := PIVOT_PARTITION(A,x);
if ik
then return BFPRT_SELECT(A[1..k],i)
else return BFPRT_SELECT(A[k+1 .. n],ik);
27.11.2003 17:18
Fundamental Algorithms
44 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
at least 3 ( 12 n5 2)
3n
10
T (n ) (1 )
7n
10
+ 6 elements.
for n CONST
n
T (n) T + . . . for n > CONST
5
n
7n
+ 6 + O(n)
c + c
5 10
n
7
c + 1 +
c n + 6 c + O(n)
5
10
9
c n + 7 c + O(n)
10
1
c n c n 7 c O (n)
10
Using the respecting constant hidden in the O(n)term, we can pick c large enough such that
7 c O(n) < 0.
1
10
cn
For that c, T (n) c n, which means that BFPRT_SELECT has a linear time complexity!
Corollary
A variant of quicksort that partitions the array around the median computed by
BFPRT_SELECT, will have a guaranteed (worst case) time complexity of O(n log n).
4. Searching
27.11.2003 17:18
Fundamental Algorithms
45 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Worst Case:
in the worst case,we will compare each element of A with x n comparisions.
The worst case applies every time when x does not occur in A!
Average case:
We assume that x does occur in A, and the probability that x = A[i] is
index i.
1
n
independent of the
If x = A[i], we will stop after i comparisons. Hence, the expected number of comparisons is
_
C (n ) =
1 n
1 n (n + 1) n + 1
i=
=
n i=1
n
2
2
27.11.2003 17:18
Fundamental Algorithms
46 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Our result is not too surprising: on average, we have to search about half of the array to find a
specific element.
Remember that this is only true if we assume that the searched element actually occurs in the
array. If we cannot be sure about that, we have to make an assumption about the probability of
finding an element in an array. This, of course, heavily depends on the scenario where searching
is needed.
Average case:
We assume that the probablity that x is in one of the intervals ] , a 1], ]a 1, a 2], . . ., ]a n 1, a n], ]
a n, + ], is exactly n +1 1 for each interval.
Then the number of comparisons performed by SORTSEARCH is
_
C (n )
1 n
(i + 1) + (n + 1)
n + 1 i = 1
1 n + 1
1 n + 2
(i + 1) 1 =
i 2
n + 1 i = 1
n + 1 i = 1
1 (n + 2) (n + 3)
n2 + 5 n + 6 4 n2 + 5 n + 2 n
2 =
n+1
2
2
2 (n + 1)
2n+ 2
27.11.2003 17:18
Fundamental Algorithms
47 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Like in an unsorted array the average number of comparison is about half the number of the
array's elements. The important difference, however, is that for the sorted array, the result also
holds if the searched element does not occur in the array.
Thus, it looks like the extra effort of sorting the array does not improve the situation too much.
However, this is not true, as we will see in the next section.
if x A[m]
then return BINSEARCH(A[p,m], x)
else return BINSEARCH (A[m+1,r]);
Solve recurrence:
> script
case n = 2 k:
We can find the solution by iteration:
n
n
n
C (n) = C + 1 = C + 2 = . . . = C k + k = C (1) + k = k + 1
2
4
2
As k = log 2(n), we get C (n) = 1 + log 2(n).
case 2 k 1 < n 2 k:
27.11.2003 17:18
Fundamental Algorithms
48 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Here, we may use that the number of comparisons is monotonously increasing with the
number of elements in the array (on strict terms, we would have to prove this . . .), thus C (n)
C (2 k) = k + 1.
As k = log 2 (n), we get C (n) 1 + log 2 (n).
As an overall result, we get:
C (n) 1 + log 2 (n) O(log n) for all n .
Remarks:
What happens if we have to insert/delete elements in our sequence?
resorting of the sequence required (possibly requires O(n log n) work).
Searching is therefore closely related to chosing appropriate data structures for inserting
and deleting elements!
Data Structure:
Bin Tree := empty Tree
 (key : integer;
leftson : BinTree;
rightson : BinTree;
);
Example:
x = (4, (2, emptyTree, emptyTree), (3, emptyTree, (5, emptyTree, em
Graph of x:
Notation:
x.key = 4
x.leftSon = (2, emptyTree, emptyTree)
x.rightSon.key = 3
27.11.2003 17:18
Fundamental Algorithms
49 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
A binary tree x is called a binary search tree, if it satisfies the following properties:
for all keys l that are sorted in x.leftSon, l x.key;
for all keys r that are sorted in x.rightSon, r x.key;
both, x.leftSon, and x.rightSon, are again binary search trees.
Examples:
y = (3, (2, emptyTree, emptyTree),(4, emptyTree,(5, emptyTree, empt
Graph of y:
if x.key > k
then return TREE_SEARCH(x.leftSon, k)
else return TREE_SEARCH(x.rightSon, k);
TREE_SEARCH returns a subtree of x that contains the value k in its top node. If k does not
occur as a key value in x, then TREE_SEARCH returns an empty tree.
return t;
27.11.2003 17:18
Fundamental Algorithms
50 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Remarks:
For a fully balanced binary tree, we know that h O(log n) (n the number of keys stored in
the tree).
Thus, TREE_SEARCH will perform O(log n) comparisons to find a certain key in a balanced
binary tree that contains n nodes/keys.
The main problem will be to build (and maintain) a balanced search tree.
if x = emptyTree
then
x := (k, emptyTree, emptyTree);
else
if x < x.key
then TREE_INSERT(x.leftSon, k)
else TREE_INSERT(x.rightSon, k);
. . . and Deleting
While we can always add additional nodes to a binary tree, we can not simply remove nodes
from a binary tree without destroying it:
if the required node has only one nonempty subtree, we can delete the top node by
replacing the tree by the repective subtree (if both subtrees are empty, the tree is replaced
by an empty subtree).
if the respective node has two nonempty subtrees, we can not delete the node; instead we
27.11.2003 17:18
Fundamental Algorithms
51 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
if x.leftSon = emptyTree
then {
// we've found the leftmost node:
k := x.key;
x := x.rightSon;
return k;
}
else
return TREE_DEL_LEFTMOST (x.leftSon);
TREE_DEL_TOPNODE (x:BinTree) {
// assume that x is nonempty
if x.rightSon = emptyTree
then
x = x.leftSon
else {
// delete leftmost node of the right son (and memorize it)
k = TREE_DEL_LEFTMOST (x.rightSon);
// make the memorized node the new top node
x.key = k;
}
}
27.11.2003 17:18
Fundamental Algorithms
52 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Examle:
27.11.2003 17:18
Fundamental Algorithms
53 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Maximal number:
A binary tree that is completely filled has a height balance of 0 for every node it is an AVL
tree.
We already know that such a completely filled binary tree has 2 h 1 nodes.
Minmal number:
A "minimal" AVL tree of height h consists of
a root node
one subtree that is a minimal AVL tree of height h 1;
one subtree that is a minmal AVL tree of height h 2;
N AVL, min(h) = 1 + N AVL, min(h 1) + N AVL, min(h 2)
In addition, we know that
a minimal AVL tree of height 1 has 1 node: N AVL, min(1) = 1
a minimal AVL tree of height 2 has 2 nodes: N AVL, min(2) = 2
This leads to the following recurrence:
N AVL, min(1) = 1
N AVL, min(2) = 2
N AVL, min(h) = 1 + N AVL, min(h 1) + N AVL, min(h 2)
We compare this recurrence with that of the Fibonacci numbers:
27.11.2003 17:18
Fundamental Algorithms
54 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
f0 = 1
f1 = 1
fh = fh1 + fh2
Let's list the first couple of values:
h
fh
1
1
2
2
3
3
4
5
5
8
6
13
7
21
8
34
N AVL, min(h)
12
20
33
54
Proof by induction:
case h = 1:
N AVL, min(1) = 1, and f 1 +1 1 = 2 1 = 1
case h = 2:
N AVL, min(2) = 2, and f 2 +1 1 = 3 1 = 2
induction step h 1 h:
N AVL, min(h)
1 + ( f h 1 + 1 1) + ( f h 2 + 1 1)
fh + fh1 1
fh+1 1
Corollaries:
h+1
2
Therefore, an AVL tree of height h will have at least 2 2 1 nodes, and at most
2 h + 1 1 nodes.
As a consequence, an AVL tree that contains n nodes will be of height (log n)
.
2. Searching in an AVL tree has a time complexity of (log n).
3. Inserting, or deleting a single element in an AVL tree has a time complexity of
(log n).
BUT: standard inserting/deleting will probably destroy the AVL property.
Fundamental Algorithms
55 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
The algorithms to search, insert, or delete elements in an AVL tree are, in principal, the same as
those on "standard" binary trees. However, the deletion, or insertion of nodes into former AVL
trees might destroy there AVL property. The resulting loss of balance woulda, as a consequence,
slow down the algorithms.
To restore the AVL property in a certain node of a binary tree, we will discuss four socalled
rotation operators:
left rotation,
right rotation,
leftright rotation, and
rightleft rotation
AVL_LEFTROT (x:BinTree) {
x := ( x.rightSon.key,
(x.key, x.leftSon, x.rightSon.leftSon),
x.rightSon.rightSon
);
27.11.2003 17:18
Fundamental Algorithms
56 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
AVL_RIGHTROT (x:BinTree) {
x := ( x.leftSon.key,
x.leftSon.leftSon,
(x.key, x.leftSon.rightSon, x.rightSon)
);
27.11.2003 17:18
Fundamental Algorithms
57 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
The combination of a right rotation on the right subtree, followed by a left rotation is generally
referred to as a rightleft rotation :
AVL_RIGHTROT_LEFTROT (x:BinTree) {
AVL_RIGHTROT(x.rightSon);
AVL_LEFTROT(x);
The effect of the two rotations can also be achieved by the following algorithm:
AVL_RIGHTLEFTROT (x:BinTree) {
x := ( x.rightSon.leftSon.key,
( x.key,
x.leftSon,
x.rightSon.leftSon.leftSon ),
( x.rightSon.key,
x.rightSon.leftSon.rightSon,
x.rightSon.rightSon)
);
27.11.2003 17:18
Fundamental Algorithms
58 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Not suprisingly, the combination of a left rotation on the left subtree, followed by a right rotation is
generally referred to as a leftright rotation.
Again, we can either implement the roation based on the combination of two "simple" rotation, or
choose the direct approach:
AVL_LEFTROT_RIGHTROT (x:BinTree) {
AVL_LEFTROT(x.leftSon);
AVL_RIGHTROT(x);
27.11.2003 17:18
Fundamental Algorithms
59 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
}
AVL_LEFTRIGHTROT (x:BinTree) {
x := ( x.leftSon.rightSon.key,
( x.leftSon.key,
x.leftSon.leftSon,
x.leftSon.rightSon.leftSon),
( x.key,
x.leftSon.rightSon.rightSon,
x.rightSon )
);
leftright rotations and rightleft rotations will always reduce the height of the
(sub)tree by 1.
left rotations and right rotations either keep the hight unchanged, or reduce it by 1,
depending on the heights of the subtrees.
Thus, rotation operators may cause a violation of the AVL property in parent
nodes!
after inserting a single node into a previously balanced AVL tree (satifying the
AVL property), at most on rotation operation is required to restore the AVL
property.
after deleting a single node in a previously balanced AVL tree, as many as log n
operations might be necessary to restore the AVL property for all nodes (one
rotation per level, travelling upwards in the tree).
after both, inserting and deleting, the possible violation of the AVL property can
occur on a much higher level than where the node was inserted/deleted. Thus, the
AVL property has to checked in the entire branch of the tree, up to the root.
27.11.2003 17:18
Fundamental Algorithms
60 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
DirectAddress Table;
Assume:
the "universe" U of keys, i.e. the set of all possible keys, is reasonably small, for example U
= {0, 1, . . ., m 1}.
Idea: "directaddress table"
Store every object in the Table element given by the object's key.
DIR_ADDR_INSERT(T:Table, x:Object) {
T[X.key] := x;
}
DIR_ADDR_DELETE(T:Table, x:Object){
T[X.key] := NIL;
}
DIR_ADDR_SEARCH(T:Table, key:Integer){
return T[key];
}
Positive:
+ very fast: every operation is O(1).
27.11.2003 17:18
Fundamental Algorithms
61 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Negative:
 all keys need to be distinct (they better should be, anyway . . .)
 m has to be small (which is a quite severe restriction!)
 if only few elements are stored, lots of Table elements are unused (waste of memory).
4.3.2. Hashing
If the universe of keys is very large, directaccess tables become impractical. However, if we can
find an easy function that
computes the table index from the key of an object,
has a relatively small range of values, and
can be computed efficiently,
we can use this function to calculate the table index.
The respective function is called a hash function, and the table is called a hash table.
The algorithms for inserting, deleting, and searching in such a simple hash table are quite
straightforward:
SMPL_HASH_INSERT(T:Table, x:Object) {
T[h(x.key)] := x;
}
SMPL_HASH_DELETE(T:Table, x:Object) {
T[h(x.key)]:= NIL;
}
SMPL_HASH_SEARCH(T:Table, x:Object) {
return T[h(x.key)];
}
Positive:
+ still very fast: O(1).
+ size of the table can be chosen freely, provided there is an appropriate hash function h.
Negative:
 values of h have to be distinct for all keys
In general, it is not possible to find a function h that generates distinct values for all possible keys
(the whole idea was to make the range of h much smaller then that of the key values . . .)
we have to deal with collisions, i.e. situations where different objects with different
keys share a common value of the hash function, and would therefore have to be stored
in the same table element.
27.11.2003 17:18
Fundamental Algorithms
62 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Algorithms:
The resulting algorithms for inserting, deleting, and searching are based on that for the
containers:
Positive:
+ hash function no longer has to return distinct vaues
+ still very fast, if the lists are short
Negative:
 deleting/searching is O(l), l the number of elements in the accessed list.
 Worst case: All elements have to be stored in one single list (very unlikey).
27.11.2003 17:18
Fundamental Algorithms
63 of 67
contains n elements; =
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
n
m
Corollary:
In the worst case, also a successful search will have to check all keys that are stored in
a hash table slot.
Therefore, the average complexity of a successful search will also be O(1 + ).
Fundamental Algorithms
64 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Towstep method:
1. multiply k by a constant (0 < < 1), and extract the fractional part of k .
2. multiply frational part by m, and use the integer part of the result as hash value.
Thus
h(k) = m ( k mod 1)
Remarks:
value of m is not critical (size of hash table can be chosen independent of hash function).
still has to be chosen wisely (see Knuth).
Efficient Implementation:
Choose m to be a power of 2, for example m = 2 p, and use a fixedpoint arithmetic with w bits for
the fractional parts (w typically being the wordlength). Then:
multiply k by 2 p (the latter term is a wbit integer);
the result is a (2 k )bit value r 1 2 w + r 0, where r 0 is the binary representation of the fractional
part of k.
multiplying r 1 2 w + r 0 by 2 p would shift the p most significant bits of r 0 into the "integer" part.
take p most significant bits of r 0 as the hash value.
Thus, the proposed multiplication method can be implemented without the need for floatingpoint
arithmetics (although fractional values are involved). The required arithmetics shifts and modulo
operations can be implemented very efficiently on most platforms, so the cost of computing the
hash value for a single key is dominated by that of one (long) integer multiplication.
27.11.2003 17:18
Fundamental Algorithms
65 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
To resolve collisions, open addressing has to allow more than one position for a specific object.
The hash function is therefore modified to generate sequences of hash table indices:
h : U{0, 1, . . ., m 1} {0, 1, . . ., m 1}
For every key k, the sequence
h(k, 0), h(k, 1), h(k, 2), . . ., h(k, m 1)
is called the probe sequence.
General approach:
An object will be stored in the first empty slot that is specified by the probe sequence.
To guarantee that an empty slot in the hash table will eventually be found, the probe
sequence [h(k, 0), . . ., h(k, m 1)] should be a permutation of [0, . . ., m 1]
Algorithms:
HASH_INSERT(T:Table, x:Object) : Integer {
for i from 0 to m1 do {
j := h(x.key,i);
if T[j]=NIL
then {
T[j] := x;
return j; /* and terminate */
}
}
cast error "hash table overflow"
}
HASH_SEARCH(T:Table, k:Integer) : Object {
i := 0;
while T[h(k,i)] != NIL and i < m {
if k = T[h(k,i)].key
then return T[h(k,i)]; /* and terminate */
i := i+1;
}
return NIL;
}
Linear Probing:
Use hash function
27.11.2003 17:18
Fundamental Algorithms
66 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
checked;
if this is still occupied, we wrap around to slots T[0], T[1], . . ., T[h 0(k) 1].
Quadratic probing:
Use hash function
h(k, i) = (h 0 (k) + c 1 i + c 2 i 2) mod m
where h 0 is an ordinary hash function, and c 1 and c 2 are suitable constants.
Double hashing:
Double hashing uses a function
h(k, i) = (h 0 (k) + i h 1 (k)) mod m
where h 0 and h 1 are (auxiliary) hash functions.
Idea:
Even if h 0 generates the same hash values for several keys, h 1 will generate different probe
sequences for any of these keys.
Choosing h 0 and h 1:
h 0 and h 1 need to have different ranges of values:
h 0 : U {0, . . ., m 0 1}, m 0 = m being the size of the hash table.
h 1(k) must never be 0 (otherwise, no probe sequence is generated)
h 1(k) should be prime to m 0 = m (for all k):
if h 1(k) and m 0 = m have a greatest common divisor d > 1, the generated probe sequence
will only examine 1d th of the hash slots.
Possible choices:
27.11.2003 17:18
Fundamental Algorithms
67 of 67
http://www.cse.tum.de/vtc/FundAlg/print/print.xml
Let m 0 = m be a power of 2, and design h 1 such that it will generate odd numbers only.
Let m 0 = m be a prime number, and let h 1 : U {1, . . ., m 1}, where m 1 < m
27.11.2003 17:18