Sorting Lower Bounds Linear Time

Sorting: Lower Bounds and Linear Time
Last Revision: September 8, 2017
Sorting: Lower Bounds and Linear Time Last Revision: September

1 /8,36201
Lower Bound for Sorting
All sorting algorithms we seen so far are based on comparing

elements
E.g., Insertion Sort, Mergesort, Heapsort and Quicksort

2 /8,36201

elements
Insertion Sort and Quicksort have worst-case running times
Θ(n2 ), while the others have worst-case running time
Θ(n log n)

2 /8,36201

elements
Θ(n log n)
Question
Can we do better?

2 /8,36201

elements
Θ(n log n)
Question
Can we do better?
Goal
We will prove that any comparison-based sorting algorithm has a
worst-case running time Ω(n log n).

2 /8,36201
Decision-tree Example
Sort < a1 , a2 , . . . , an > a1 : a2

≤ >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1

3 /8,36201
Sort < a1 , a2 , . . . , an > a1 : a2

≤ >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1
Each internal node is labeled ai : aj for {1, 2, . . . , n}

3 /8,36201
Sort < a1 , a2 , . . . , an > a1 : a2

≤ >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1

The left subtree shows subsequent comparisons if ai ≤ aj

3 /8,36201
Sort < a1 , a2 , . . . , an > a1 : a2

≤ >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1

The right subtree shows subsequent comparisons if ai > aj

3 /8,36201
Sort < a1 , a2 , . . . , an > a1 : a2

≤ >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1

Each leaf corresponds to an input ordering

3 /8,36201
Sort < a1 , a2 , a3 > a1 : a2

=< 6, 8, 5 >: 6≤8 >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1


4 /8,36201
Sort < a1 , a2 , a3 > a1 : a2

=< 6, 8, 5 >: ≤ >
a2 : a3 a1 : a3
≤ 8>5 ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1


5 /8,36201
Sort < a1 , a2 , a3 > a1 : a2

=< 6, 8, 5 >: ≤ >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ 6>5 ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1


6 /8,36201
Sort < a1 , a2 , a3 > a1 : a2

=< 6, 8, 5 >: ≤ >
a2 : a3 a1 : a3
≤ > ≤ >
a1 , a2 , a3 a1 : a3 a2 , a1 , a3 a2 : a3
≤ > ≤ >
a1 , a3 , a2 a3 , a1 , a2 a2 , a3 , a1 a3 , a2 , a1
5≤6≤8


7 /8,36201
Decision-tree Model
A decision tree can model the execution of any comparison-based

sorting algorithm

8 /8,36201
Decision-tree Model

sorting algorithm
One tree for each input size n

8 /8,36201
Decision-tree Model

sorting algorithm
One tree for each input size n
Worst-case running time = height of tree

8 /8,36201
Theorem
Any comparison-based sorting algorithm requires Ω(n log n)
comparisons in the worst case.

9 /8,36201
Theorem
Proof.
A decision tree to sort n elements must have at least n!
leaves, since each of the n! orderings is a possible answer.

9 /8,36201
Theorem
Proof.
A binary tree of height h has at most 2h leaves

9 /8,36201
Theorem
Proof.
Thus, n! ≤ 2h
⇒ h ≥ log n! = Ω(n log n) (Stirling’s approximation)

9 /8,36201
Theorem
Proof.
Thus, n! ≤ 2h
⇒ h ≥ log n! = Ω(n log n) (Stirling’s approximation)
Corollary
Heapsort and merge sort are asymptotically optimal
comparison-based sorting algorithms.
9 /8,36201
Lower Bound for Average Running Time of Sorting
We just proved that worst case number of comparisons used is

Ω(n log n)
Suppose that each of the n! input permutations is equally
likely. What can be said about the average case running time?
Note: Average is taken by adding up individual running time of

algorithm on each possible input and dividing total by n!.

10 /8,36201
We just proved that worst case number of comparisons used is

Ω(n log n)
Suppose that each of the n! input permutations is equally
likely. What can be said about the average case running time?
Note: Average is taken by adding up individual running time of

algorithm on each possible input and dividing total by n!.
Theorem
When all input permutations are equally likely, any
comparison-based sorting algorithm requires Ω(n log n)
comparisons on average.

10 /8,36201
Theorem
Proof.
The External Path Length (EPL) of a tree is the sum over all leaves
of the tree, of the length of the paths from the root to the leaves.

11 /8,36201
Theorem
Proof.
Average number of comparisons used by a sorting algorithm is EPL
of its associated comparison tree divided by n!.

11 /8,36201
Theorem
Proof.
The EPL of a binary tree with m leaves is at least m log2 m + O(m).
The comparison tree has m = n! leaves
⇒ its external path length is n! log2 (n!) + O(n!)
⇒ average number of comparisons used is log2 (n!) + O(1).

11 /8,36201
Theorem
Proof.
The EPL of a binary tree with m leaves is at least m log2 m + O(m).
The comparison tree has m = n! leaves
⇒ its external path length is n! log2 (n!) + O(n!)
⇒ average number of comparisons used is log2 (n!) + O(1).
We already saw log2 (n!) = Ω(n log n).

11 /8,36201
Can we do better?
Are there sorting algorithms which are not comparison-based?

Can they beat the Ω(n log n) lower bound?

12 /8,36201
Can we do better?

Counting sort
Assumes items are in set {1, 2, . . . , k}.
Is a stable sort (defined soon).

12 /8,36201
Can we do better?

Counting sort
Assumes items are in set {1, 2, . . . , k}.
Is a stable sort (defined soon).
Radix sort
Assumes items are stored in fixed size words using finite
alphabet

12 /8,36201
Counting Sort
Counting-sort(A, B, k)
Input: A[1 . . . n], where A[j] ∈ {1, 2, . . . , k}
Output: B[1 . . . n], sorted
let C [1 . . . k] be a new array;
for i ← 1 to k do
C [i] ← 0;
end
for j ← 1 to n do
C [A[j]] ← C [A[j]] + 1; // C [i] = |{key = i}|
end
for i ← 2 to k do
C [i] ← C [i] + C [i − 1]; // C [i] = |{key ≤ i}|
end
for j ← n to 1 do
B[C [A[j]]] ← A[j];
C [A[j]] ← C [A[j]] − 1;
end
13 /8,36201
Example: Counting Sort
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C

14 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 0 0 0 0
for i ← 1 to k do
C[i] ← 0;
end
Count # of instances of each value.

15 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 0 0 0 1
for j ← 1 to n do
C[A[j]] ← C[A[j]] + 1; // C[i] = |{key = i}|
end

16 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 0 1 0 1
for j ← 1 to n do
C[A[j]] ← C[A[j]] + 1; // C[i] = |{key = i}|
end

17 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 1 0 1
for j ← 1 to n do
C[A[j]] ← C[A[j]] + 1; // C[i] = |{key = i}|
end

18 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 1 0 2
for j ← 1 to n do
C[A[j]] ← C[A[j]] + 1; // C[i] = |{key = i}|
end
Count # of values ≤ current value.

19 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 2 0 2
for j ← 1 to n do
C[A[j]] ← C[A[j]] + 1; // C[i] = |{key = i}|
end

20 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 2 0 2
B C0 1 3 0 2
for i ← 2 to k do
C[i] ← C[i] + C[i − 1]; // C[i] = |{key ≤ i}|
end

21 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 2 0 2
B C0 1 3 3 2
for i ← 2 to k do
C[i] ← C[i] + C[i − 1]; // C[i] = |{key ≤ i}|
end

22 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 2 0 2
B C0 1 3 3 5
for i ← 2 to k do
C[i] ← C[i] + C[i − 1]; // C[i] = |{key ≤ i}|
end

23 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 3 3 5
B 2 C0 1 2 3 5
for j ← n to 1 do
B[C[A[j]]] ← A[j];
C[A[j]] ← C[A[j]] − 1;
end
Place items properly.

Walk through A from right to left. When scanning x = A[j], C 0 [x] will
hold appropriate location in B of the rightmost currently unplaced x.
24 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 3 3 5
B 2 4 C0 1 2 3 4
for j ← n to 1 do
B[C[A[j]]] ← A[j];
C[A[j]] ← C[A[j]] − 1;
end

25 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 3 3 5
B 1 2 4 C0 0 2 3 4
for j ← n to 1 do
B[C[A[j]]] ← A[j];
C[A[j]] ← C[A[j]] − 1;
end

26 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 3 3 5
B 1 2 2 4 C0 0 1 3 4
for j ← n to 1 do
B[C[A[j]]] ← A[j];
C[A[j]] ← C[A[j]] − 1;
end

27 /8,36201
1 2 3 4 5 1 2 3 4
A 4 2 1 4 2 C 1 3 3 5
B 1 2 2 4 4 C0 0 1 3 3
for j ← n to 1 do
B[C[A[j]]] ← A[j];
C[A[j]] ← C[A[j]] − 1;
end

28 /8,36201
Analysis
for i ← 1 to k do
C [i] ← 0; // O(k)
end

29 /8,36201
Analysis
for i ← 1 to k do
C [i] ← 0; // O(k)
end
for j ← 1 to n do
C [A[j]] ← C [A[j]] + 1; // O(n)
end

29 /8,36201
Analysis
for i ← 1 to k do
C [i] ← 0; // O(k)
end
for j ← 1 to n do
C [A[j]] ← C [A[j]] + 1; // O(n)
end
for i ← 2 to k do
C [i] ← C [i] + C [i − 1]; // O(k)
end

29 /8,36201
Analysis
for i ← 1 to k do
C [i] ← 0; // O(k)
end
for j ← 1 to n do
C [A[j]] ← C [A[j]] + 1; // O(n)
end
for i ← 2 to k do
C [i] ← C [i] + C [i − 1]; // O(k)
end
for j ← n to 1 do
B[C [A[j]]] ← A[j];
C [A[j]] ← C [A[j]] − 1; // O(n)
end

29 /8,36201
Analysis
for i ← 1 to k do
C [i] ← 0; // O(k)
end
for j ← 1 to n do
C [A[j]] ← C [A[j]] + 1; // O(n)
end
for i ← 2 to k do
C [i] ← C [i] + C [i − 1]; // O(k)
end
for j ← n to 1 do
B[C [A[j]]] ← A[j];
C [A[j]] ← C [A[j]] − 1; // O(n)
end
Total: O(n + k)
29 /8,36201
Running Time
If k = O(n), then counting sort takes O(n) time.

30 /8,36201
Running Time
But didn’t we prove that sorting must take Ω(n log n) time?

30 /8,36201
Running Time
No, actually we proved that any comparison-based sorting

algorithm takes Ω(n log n) time.

30 /8,36201
Running Time

Note that counting sort is not a comparison-based sorting

algorithm.

30 /8,36201
Running Time

Note that counting sort is not a comparison-based sorting

algorithm.
In fact, it makes no comparisons at all!

30 /8,36201
Stable Sorting
Counting sort is a stable sort

it preserves the input order among equal elements.
4 2 1 4 2
1 2 2 4 4

31 /8,36201
Stable Sorting
Counting sort is a stable sort

it preserves the input order among equal elements.
4 2 1 4 2
1 2 2 4 4
Exercise
What other sorts have this property?

31 /8,36201
Radix Sort
Input is array A of n numbers.

Each number is represented by d digits.
Digits have values in {0, 1, , k − 1}.
Radix Sort: Sort the numbers digit (column) by digit, starting
with least significant (rightmost) digit and moving to the
most significant digit (rightmost)

32 /8,36201
Radix Sort
Input is array A of n numbers.

Each number is represented by d digits.
Digits have values in {0, 1, , k − 1}.
Radix Sort: Sort the numbers digit (column) by digit, starting
with least significant (rightmost) digit and moving to the
most significant digit (rightmost)
2 32 9 272 0 272 0 2 32 9 2 32 9
5 45 7 535 5 232 9 5 35 5 2 72 0
3 65 7 343 6 343 6 3 43 6 3 43 6
5 83 9 545 7 583 9 5 45 7 3 65 7
3 43 6 365 7 535 5 3 65 7 5 35 5
2 72 0 232 9 545 7 2 72 0 5 45 7
5 35 5 583 9 365 7 5 83 9 5 83 9
32 /8,36201
Radix Sort: Correctness
Induction on digit position

Assume that the numbers are
sorted by their low-order i − 1
digits 272 0 2 32 9
232 9 5 35 5
Sort on digit i 343 6 3 43 6
583 9 5 45 7
535 5 3 65 7
545 7 2 72 0
365 7 5 83 9

33 /8,36201

digits 272 0 2 32 9
232 9 5 35 5
Two numbers that differ on 583 9 5 45 7
digit i are correctly sorted by 535 5 3 65 7
their low-order i digits
545 7 2 72 0
365 7 5 83 9

34 /8,36201

digits 272 0 2 32 9
232 9 5 35 5
Two numbers that differ on 583 9 5 45 7
digit i are correctly sorted by 535 5 3 65 7
their low-order i digits
545 7 2 72 0
Two numbers equal on digit i 365 7 5 83 9
are put in the same order as
the input ⇒ correctly sorted
by their low-order i digits

35 /8,36201
Radix Sort: Running Time & Application
Lemma
Given n d-digit numbers in which each digit can take on up to k
possible values, radix sort correctly sorts these numbers in
O(d(n + k)) time if the stable sort it uses takes O(n + k) time.

36 /8,36201
Lemma
Application:
Sorting numbers in the range from 0 to nb − 1, where b is a
constant

36 /8,36201
Lemma
Application:
constant
b log n bits for each number

36 /8,36201
Lemma
Application:
constant
each number can be viewed as having O(b) digits of log n bits
each

36 /8,36201
Lemma
Application:
constant
each
running time is O(d(n + k)) = O(b(n + 2log n )) = O(bn)

36 /8,36201
Lemma
Application:
constant
each
running time is O(d(n + k)) = O(b(n + 2log n )) = O(bn)
since b is a constant, the running time is O(n).

36 /8,36201

Sorting Lower Bounds Linear Time

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sorting Lower Bounds Linear Time

Uploaded by

Copyright:

Available Formats

Sorting: Lower Bounds and Linear Time

Last Revision: September 8, 2017

Sorting: Lower Bounds and Linear Time Last Revision: September

All sorting algorithms we seen so far are based on comparing

Sorting: Lower Bounds and Linear Time Last Revision: September

All sorting algorithms we seen so far are based on comparing

Sorting: Lower Bounds and Linear Time Last Revision: September

All sorting algorithms we seen so far are based on comparing

Sorting: Lower Bounds and Linear Time Last Revision: September

All sorting algorithms we seen so far are based on comparing

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , . . . , an > a1 : a2

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , . . . , an > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , . . . , an > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , . . . , an > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , . . . , an > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , a3 > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , a3 > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , a3 > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

Sort < a1 , a2 , a3 > a1 : a2

Each internal node is labeled ai : aj for {1, 2, . . . , n}

Sorting: Lower Bounds and Linear Time Last Revision: September

A decision tree can model the execution of any comparison-based

Sorting: Lower Bounds and Linear Time Last Revision: September

A decision tree can model the execution of any comparison-based

One tree for each input size n

Sorting: Lower Bounds and Linear Time Last Revision: September

A decision tree can model the execution of any comparison-based

One tree for each input size n

Worst-case running time = height of tree

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

We just proved that worst case number of comparisons used is

Note: Average is taken by adding up individual running time of

Sorting: Lower Bounds and Linear Time Last Revision: September

We just proved that worst case number of comparisons used is

Note: Average is taken by adding up individual running time of

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

Sorting: Lower Bounds and Linear Time Last Revision: September

Are there sorting algorithms which are not comparison-based?

Sorting: Lower Bounds and Linear Time Last Revision: September

Are there sorting algorithms which are not comparison-based?