Chapter 7-8 PDF

1
Data Structures and Algorithms
BITS-Pilani K. K. Birla Goa Campus
1
Material for the presentation taken from Cormen, Leiserson, Rivest and
Stein, Introduction to Algorithms, Third Edition; 1
BITS-Pilani K. K. Birla Goa Campus Data Structures and Algorithms
Ch. 7 : Quicksort
I Average case running time : O(n lg n)
2
Ch. 7 : Quicksort

I Constant factor associated with O(n lg n) is small compared
to merge sort.
2
Ch. 7 : Quicksort

to merge sort. (Best practical choice.)
2
Ch. 7 : Quicksort

I Worst case running time : O(n2 )
2
Ch. 7 : Quicksort

I Worst case running time : O(n2 )
I Uses the divide-and-conquer paradigm
2
Quicksort
3
Quicksort : Partition
4
I A = [ 4, 2, 5, 1, 3 ] , p = 1 , r = 5
4
5
6
PARTITION procedure: four regions
7
7
Running time : Θ(n)
7
Quicksort
[ 2, 5, 1, 4, 3 ]
8
Quicksort
[ 2, 5, 1, 4, 3 ]
[ 2, 1, 3, 4, 5 ]
8
Quicksort
[ 2, 5, 1, 4, 3 ]
[ 2, 1, 3, 4, 5 ]
[ 2, 1 ] [ 3 ] [ 4, 5 ]
8
Quicksort
[ 2, 5, 1, 4, 3 ]
[ 2, 1, 3, 4, 5 ]
[ 2, 1 ] [ 3 ] [ 4, 5 ]
[ 1, 2 ] [ 3 ] [ 4, 5 ]
8
Quicksort
[ 2, 5, 1, 4, 3 ]
[ 2, 1, 3, 4, 5 ]
[ 2, 1 ] [ 3 ] [ 4, 5 ]
[ 1, 2 ] [ 3 ] [ 4, 5 ]
[ 1 ] [ 2 ] [ 3 ] [ 4, 5 ]
8
Quicksort
[ 2, 5, 1, 4, 3 ]
[ 2, 1, 3, 4, 5 ]
[ 2, 1 ] [ 3 ] [ 4, 5 ]
[ 1, 2 ] [ 3 ] [ 4, 5 ]
[ 1 ] [ 2 ] [ 3 ] [ 4, 5 ]
[1] [2] [3] [4] [5]
8
Performance of quicksort
I Running time depends on whether the partition is balanced.
9

I Worst-case partitioning
9

T (n) = T (n − 1) + T (0) + Θ(n)
9

T (n) = T (n − 1) + T (0) + Θ(n)

= T (n − 1) + Θ(n)
9

T (n) = T (n − 1) + T (0) + Θ(n)

= T (n − 1) + Θ(n)
We can use substitution method : T (n) = Θ(n2 )
9

T (n) = T (n − 1) + T (0) + Θ(n)

= T (n − 1) + Θ(n)
I Best-case partitioning
9

T (n) = T (n − 1) + T (0) + Θ(n)

= T (n − 1) + Θ(n)
T (n) = 2T (n/2) + Θ(n)
9

T (n) = T (n − 1) + T (0) + Θ(n)

= T (n − 1) + Θ(n)
T (n) = 2T (n/2) + Θ(n)
We can use master method : T (n) = Θ(n lg n)
9

T (n) = T (n − 1) + T (0) + Θ(n)

= T (n − 1) + Θ(n)
T (n) = 2T (n/2) + Θ(n)
We can use master method : T (n) = Θ(n lg n)
I What would be the average-case performance?

9
Average-case performance
I Average case performance will be closer to the best-case

performance
10

performance
I Suppose 9-to-1 proportional split
10

performance
I Suppose 9-to-1 proportional split
T (n) = T (9n/10) + T (n/10) + cn
10
9-to-1 proportional split
11
I Though a 9-to-1 split seems unbalanced, the running time is

O(n lg n).
12

O(n lg n).
I What if the split was 99-to-1? What would be the running
time?
12

O(n lg n).
time?
I Any split of the form rn and (1 − r )n, where r is between 0
and 1, will give a running time of O(n lg n).
12

O(n lg n).
time?
I Any split of the form rn and (1 − r )n, where r is between 0
and 1, will give a running time of O(n lg n).
I It is unlikely that at each level PARTITION procedure will give
us a highly unbalanced split (assuming a random input array).
12
Randomized version of quicksort
I We will uniformly randomly select an element to be the pivot

element.
13
Randomized version of quicksort
I We will uniformly randomly select an element to be the pivot

element.
I This should lead to the split produced by the PARTITION
procedure to be well balanced on average.
13
Randomized Quicksort
14
Randomized Quicksort
14
Expected running time of randomized quicksort
15
I How many times can the same element be selected as a pivot

during the entire run of the quicksort algorithm?
15

I Therefore, the partition procedure will be called at most n
times.
15

I Therefore, the partition procedure will be called at most n
times.
I So, the overall running time can be bounded by the number of
times line 4 of the partition procedure is executed.
15
I Lemma : Let X be the number of comparisons performed in

line 4 of the PARTITION procedure. Then the running time
of quicksort is O(n + X ).
16

I We will derive an overall bound on the total number of
comparisons X .
16

comparisons X .
I We will rename the elements of A as z1 , z2 , . . . , zn , with zi
being the ith smallest element.
16

comparisons X .
I Also, we will let Zij denote the set {zi , zi+1 , . . . , zj }
16

comparisons X .
I Also, we will let Zij denote the set {zi , zi+1 , . . . , zj }
I Claim : Each pair of elements are compared at most once.
16
I Indicator random variable:

Xij = I {zi is compared to zj }
17
I Indicator random variable:

Xij = I {zi is compared to zj }
I Total number of comparisons X :
n−1 X
X n
X = Xij
i=1 j=i+1
17
Expected number of comparisons
I Expected value of the total number of comparisons E [X ]:

 
Xn−1 X n
E [X ] = E  Xij 
i=1 j=i+1
18

 
Xn−1 X n
E [X ] = E  Xij 
i=1 j=i+1
n−1 X
X n
= E [Xij ]
i=1 j=i+1
18

 
Xn−1 X n
E [X ] = E  Xij 
i=1 j=i+1
n−1 X
X n
= E [Xij ]
i=1 j=i+1
n−1 X
X n
= Pr {zi is compared to zj }
i=1 j=i+1
18
I RANDOMIZED-PARTITION procedure chooses each pivot

randomly and independently.
19
I RANDOMIZED-PARTITION procedure chooses each pivot

randomly and independently.
I Let us think: when will two elements not be compared?
19
I Once pivot x is chosen such that zi < x < zj , we know that zi

and zj cannot be compared at any subsequent time.
20

I On the other hand, if zi is chosen as a pivot before any other
element in Zij , then zi will be compared with all the elements
in Zij .
20

I On the other hand, if zi is chosen as a pivot before any other
element in Zij , then zi will be compared with all the elements
in Zij .
I Similarly, if zj is chosen as a pivot before any other element in
Zij , then zj will be compared with all the elements in Zij .
20
I In any run of the randomized quicksort algorithm, all elements

in the set Zij are equally likely to be chosen as the first pivot
from the set.
21
Probability that zi is compared to zj
22
23
24
25
Expected value of the total number of comparisons E [X ]:

n−1 X
X n
E [X ] = Pr {zi is compared to zj }
i=1 j=i+1
26

n−1 X
X n
i=1 j=i+1
n−1 X
n
X 2
=
j −i +1
i=1 j=i+1
26

n−1 X
X n
i=1 j=i+1
n−1 X
n n−1 X
n−i
X 2 X 2
= =
j −i +1 k +1
i=1 j=i+1 i=1 k=1
26

n−1 X
X n
i=1 j=i+1
n−1 X
n n−1 X
n−i
X 2 X 2
= =
j −i +1 k +1
i=1 j=i+1 i=1 k=1
n−1 n
X X 2
<
k
i=1 k=1
26

n−1 X
X n
i=1 j=i+1
n−1 X
n n−1 X
n−i
X 2 X 2
= =
j −i +1 k +1
i=1 j=i+1 i=1 k=1
n−1 n
X X 2
<
k
i=1 k=1
n−1 n
X X 1
= O(lg n) ( Using = O(lg n) )
k
i=1 k=1
26

n−1 X
X n
i=1 j=i+1
n−1 X
n n−1 X
n−i
X 2 X 2
= =
j −i +1 k +1
i=1 j=i+1 i=1 k=1
n−1 n
X X 2
<
k
i=1 k=1
n−1 n
X X 1
= O(lg n) ( Using = O(lg n) )
k
i=1 k=1
= O(n lg n)
26
I Expected running time = Expected number of comparisons
27
I Expected running time = Expected number of comparisons

I Expected running time of randomized quicksort is O(n lg n)
27
Order of growth of lg(n!)
I Stirling’s approximation:
28
1
I Leading term in Stirling’s approximation is nn+ 2
28
1
I Leading term in Stirling’s approximation is nn+ 2
I lg(n!) = Θ(n lg n)
28
Ch. 8 : Sorting in Linear Time
I Comparison sort
29
I Comparison sort
I Worst case running time is at least Ω(n lg n).
29
I Comparison sort
I Is it possible to do better?
29
I Comparison sort
I Is it possible to do better? Can we have a sorting algorithm
whose worst case running time is o(n lg n) ?
29
I Comparison sort
I Is it possible to do better? Can we have a sorting algorithm
whose worst case running time is o(n lg n) ?
I We will show that any comparison sort must make at least
Ω(n lg n) comparisons in the worst case.
29
Insertion sort : comparisons
30
Merge procedure
31
Quicksort Partition procedure
32
Lower bound on number of comparisons for sorting:
Assumptions
I We will assume all input elements are distinct
33
Lower bound on number of comparisons for sorting:
Assumptions
I We will assume all input elements are distinct
I All comparisons are of the form ai ≤ aj
33
Decision tree for insertion sort
34
34
I Any correct sorting algorithm must be able to produce each of

the n! permutations of the n input elements.
34
Lower bound for the worst case
I The height of the decision tree would give us the worst-case

number of comparisons for the corresponding comparison sort.
35

I What should be the minimum height of the decision tree such
that we have at least n! leaf nodes?
35

I What should be the minimum height of the decision tree such
that we have at least n! leaf nodes?
I This minimum height would give us the lower bound on the
number of comparisons that any comparison sort must make.
35
I Theorem : Any comparison sort algorithm requires Ω(n lg n)

comparisons in the worst case.
36

Let the decision tree corresponding to the most efficient
comparison sort algorithm be of height h and have l leaf
nodes.
36

nodes.
n! ≤ l
36

nodes.
n! ≤ l ≤ 2h
36

nodes.
n! ≤ l ≤ 2h
h ≥ lg(n!)
36

nodes.
n! ≤ l ≤ 2h
h ≥ lg(n!)
≥ Θ(n lg n)
36

nodes.
n! ≤ l ≤ 2h
h ≥ lg(n!)
≥ Θ(n lg n)
= Ω(n lg n)
36

nodes.
n! ≤ l ≤ 2h
h ≥ lg(n!)
≥ Θ(n lg n)
= Ω(n lg n)
I Any comparison sort algorithm would take Ω(n lg n) time.
36
Worst case lower bounds for computational problems
I Any comparison based algorithm would take at least Ω(n lg n)

time in the worst case for solving the sorting problem.
37

I Heapsort and Merge sort are asymptotically optimal algorithm
for comparison sorting.
37

I Heapsort and Merge sort are asymptotically optimal algorithm
for comparison sorting.
37
Counting sort
I Assumes that each of the input element is an integer in the

range 0 to k for some integer k.
38
Counting sort

I Input array : A[1 . . . n]
38
Counting sort

I Output array : B[1 . . . n]
38
Counting sort

I Output array : B[1 . . . n]
I Temporary working storage : C [0 . . . k]
38
Counting sort operations
39
39
39
39
39
39
Counting sort
40
Running time of Counting sort
I Overall running time Θ(k + n)
41

I When k = O(n), the counting sort runs in Θ(n) time.
41

I Why Counting sort beats the lower bound of Ω(n lg n)?
41

I Counting sort is stable. (What is a stable sorting algorithm?)
41

I Counting sort is stable. (What is a stable sorting algorithm?)
I Which of the algorithms that we have learned are stable?
41
Sorting in linear time
I Radix sort
42
Sorting in linear time
I Radix sort
I Bucket sort
42
I
43
I
44

Chapter 7-8 PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter 7-8 PDF

Uploaded by

Copyright:

Available Formats

1

Data Structures and Algorithms

BITS-Pilani K. K. Birla Goa Campus

I Average case running time : O(n lg n)

I Average case running time : O(n lg n)

I Average case running time : O(n lg n)

I Average case running time : O(n lg n)

I Average case running time : O(n lg n)

Running time : Θ(n)

I Running time depends on whether the partition is balanced.

I Running time depends on whether the partition is balanced.

I Running time depends on whether the partition is balanced.

T (n) = T (n − 1) + T (0) + Θ(n)

I Running time depends on whether the partition is balanced.

T (n) = T (n − 1) + T (0) + Θ(n)

I Running time depends on whether the partition is balanced.

T (n) = T (n − 1) + T (0) + Θ(n)

We can use substitution method : T (n) = Θ(n2 )

I Running time depends on whether the partition is balanced.

T (n) = T (n − 1) + T (0) + Θ(n)

We can use substitution method : T (n) = Θ(n2 )

I Running time depends on whether the partition is balanced.

T (n) = T (n − 1) + T (0) + Θ(n)

We can use substitution method : T (n) = Θ(n2 )

T (n) = 2T (n/2) + Θ(n)

I Running time depends on whether the partition is balanced.

T (n) = T (n − 1) + T (0) + Θ(n)

We can use substitution method : T (n) = Θ(n2 )

T (n) = 2T (n/2) + Θ(n)

We can use master method : T (n) = Θ(n lg n)

I Running time depends on whether the partition is balanced.

T (n) = T (n − 1) + T (0) + Θ(n)

We can use substitution method : T (n) = Θ(n2 )

T (n) = 2T (n/2) + Θ(n)

We can use master method : T (n) = Θ(n lg n)

I What would be the average-case performance?

I Average case performance will be closer to the best-case

I Average case performance will be closer to the best-case

I Average case performance will be closer to the best-case

T (n) = T (9n/10) + T (n/10) + cn

I Though a 9-to-1 split seems unbalanced, the running time is

I Though a 9-to-1 split seems unbalanced, the running time is

I Though a 9-to-1 split seems unbalanced, the running time is

I Though a 9-to-1 split seems unbalanced, the running time is

I We will uniformly randomly select an element to be the pivot

I We will uniformly randomly select an element to be the pivot

I How many times can the same element be selected as a pivot

I How many times can the same element be selected as a pivot

I How many times can the same element be selected as a pivot

I Lemma : Let X be the number of comparisons performed in

I Lemma : Let X be the number of comparisons performed in

I Lemma : Let X be the number of comparisons performed in

I Lemma : Let X be the number of comparisons performed in

I Lemma : Let X be the number of comparisons performed in

I Indicator random variable:

I Indicator random variable:

I Expected value of the total number of comparisons E [X ]:

I Expected value of the total number of comparisons E [X ]:

I Expected value of the total number of comparisons E [X ]:

I RANDOMIZED-PARTITION procedure chooses each pivot

I RANDOMIZED-PARTITION procedure chooses each pivot

I Once pivot x is chosen such that zi < x < zj , we know that zi