EEE 13 LECTURE 13

SORTING ALGORITHMS
MOTIVATION

THE SORTING PROBLEM

Given a set of N items stored in a linear data structure, how
do we sort the items efficiently?

Example:

int iArr[10] = {8, 6, 2, 7, 5, 1, 3, 0, 9, 4};



//How do we generate {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}?
MOTIVATION

APPLICATIONS
‣ Sorting database records
‣ Sorting books
‣ Improving Search Speeds
‣ Finding the closest pair of numbers in a set
‣ Determine if an element is unique
‣ Frequency distribution
‣ Determine k-th largest element in a set
‣ Improving the performance of some Graph algorithms
‣ String processing
‣ Data compression
‣ Scheduling Optimization
‣ Convex Hull problem (Computational Geometry)
‣ …and anything that can benefit from a sorted list of items
REVIEW

BUBBLE SORT
Pseudocode:



repeat
hasChanged := false
repeat with index from 1 to itemCount
if ((item at index - 1) > (item at (index))
swap (item at index - 1) with (item at (index))
hasChanged := true
until hasChanged = false
REVIEW

WHAT DOES THAT LOOK LIKE IN CODE?

See bubble_sort.cpp
SORTING ALGORITHMS

WHY IS IT CALLED BUBBLE SORT?

See: https://www.youtube.com/watch?
v=70kPO1xEhBI&feature=youtu.be
SORTING ALGORITHMS

WHY DOES IT SUCK?

SORTING ALGORITHMS

WHY DOES IT SUCK?

‣ It performs too many redundant checks!

‣ Time complexity is O(N2) under normal circumstances

‣ Space complexity is O(1) auxiliary

SORTING ALGORITHMS

SELECTION SORT

index item

0 3

1 4

2 1

3 2
SORTING ALGORITHMS

SELECTION SORT
What is the lowest item from

index item
the entire list?

0 3

1 4

2 1

3 2
SORTING ALGORITHMS

SELECTION SORT
What is the lowest item from

index item
the entire list?



0 3
item 1 @ index 2

1 4

2 1

3 2
SORTING ALGORITHMS

SELECTION SORT
What is the lowest item from

index item
the entire list?



0 1
item 1 @ index 2



> Swap index 2 with index 0 1 4

2 3

3 2
SORTING ALGORITHMS

SELECTION SORT
What is the lowest item from

index item
the entire list?



0 1
item 1 @ index 2



> Swap index 2 with index 0 1 4

> index 0 is sorted, so remove
 2 3

from test
3 2
SORTING ALGORITHMS

SELECTION SORT
What is the lowest item from

index item
index 1 to index 3?



0 1
> item 2 @ index 3

1 4

2 3

3 2
SORTING ALGORITHMS

SELECTION SORT
What is the lowest item from

index item
index 1 to index 3?



0 1
> item 2 @ index 3



> Swap with index 1
 1 2


> Rinse and repeat until you
 2 3
are at the last item
3 4
SORTING ALGORITHMS

SELECTION SORT
See selection_sort.c
SORTING ALGORITHMS

COMPLEXITY ANALYSIS
‣ As terrible as bubble sort

‣ Time complexity is O(N2) even at best case

‣ Space complexity is O(1)

SORTING ALGORITHMS

HOW CAN WE DO BETTER IN TERMS OF TIME COMPLEXITY?

Recursion is useful for a strategy called

"Divide and Conquer".



The Divide and Conquer problem solving paradigm allows
us to solve large problems by dividing them into smaller
subproblems and combining their results.

A classic example of a Divide and Conquer Algorithm is

called Merge Sort.
By Swfung8 - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=14961648
SORTING ALGORITHMS

MERGE SORT PSEUDOCODE

function mergesort(m)
var list left, right, result
if length(m) ≤ 1
return m
else
var middle = length(m) / 2
for each x in m up to middle - 1
add x to left
for each x in m at and after middle
add x to right
left = mergesort(left)
right = mergesort(right)
if last(left) ≤ first(right)
append right to left
return left
result = merge(left, right)
return result
SORTING ALGORITHMS

MERGE PSEUDOCODE
function merge(left,right)
var list result
while length(left) > 0 and length(right) > 0
if first(left) ≤ first(right)
append first(left) to result
left = rest(left)
else
append first(right) to result
right = rest(right)
if length(left) > 0
append rest(left) to result
if length(right) > 0
append rest(right) to result
return result
SORTING ALGORITHMS

MERGE SORT IN C++

See merge_sort.cpp
SORTING ALGORITHMS

TIME COMPLEXITY
‣ The function merge() takes at most N operations to
combine the arrays

‣ merge_sort() however, has O(log(N)) recursive calls in it

‣ Therefore, merge_sort() has O(N*log(N)) time complexity

always
SORTING ALGORITHMS

HOW MUCH ADDITIONAL SPACE DOES MERGE SORT USE?

‣ In our implementation, we have a leftArray and rightArray
that each contain a copy of half of the input array

‣ The total size of all the arrays in a single recursion stack is N

‣ We have log(N) such rows because of the recursion stacks

depth

‣ Therefore, the space complexity of merge sort is

O(N*log(N))
SORTING ALGORITHMS

MERGE SORT
The time complexity of merge sort is significantly better
against the performance of bubble sort and selection sort,
however, that came at the cost of huge additional space
requirements

Can we do better?
SORTING ALGORITHMS

INSERTION SORT

By Swfung8 - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=14529615

SORTING ALGORITHMS

INSERTION SORT PSEUDOCODE

function insertionSort(array A)
for i from 1 to length[A]-1 do
value := A[i]
j := i-1
while j >= 0 and A[j] > value do
A[j+1] := A[j]
j := j-1
done
A[j+1] = value
done
SORTING ALGORITHMS

INSERTION SORT IN C++

See insertion_sort.cpp
SORTING ALGORITHMS

COMPLEXITY ANALYSIS OF INSERTION SORT

‣ Given a random array, it will perform O(N2) operations,
thus it has O(N2) time complexity


‣ Given that it only uses normal variables to aid it in sorting,

it only uses O(1) space
SORTING ALGORITHMS

WAIT, THAT’S BAD! WHY ARE WE EVEN DEALING WITH THIS?

‣ Selection sort and "unoptimized" bubble sort always run at
O(N2) time because they go through sorted elements
repeatedly because of their nature


‣ Insertion sort, however, has approximately Ω(N) time

complexity if the array is almost sorted (Reason:
considerably less shifting if array is almost sorted)


‣ Insertion sort is also the fastest algorithm among the three

O(N2) algorithms in empirical tests
SORTING ALGORITHMS

INSERTION SORT
‣ Now, we have a quick and memory efficient algorithm for
ideal and almost ideal cases


‣ Can we do better?
SORTING ALGORITHMS

QUICKSORT
See https://visualgo.net/sorting for animations
SORTING ALGORITHMS

PARTITIONING

18 17 4 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

18 17 4 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 18 4 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 18 4 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 4 18 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 4 18 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 4 18 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 4 18 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 4 18 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 4 18 27 19 8 22
SORTING ALGORITHMS

PARTITIONING

17 4 8 18 27 19 22
SORTING ALGORITHMS

PARTITIONING

17 4 8 18 27 19 22
SORTING ALGORITHMS

PARTITIONING

17 4 8 18 27 19 22
SORTING ALGORITHMS

PARTITIONING

17 4 8 18 27 19 22
SORTING ALGORITHMS

PARTITIONING

4 8 17 18 19 22 27
SORTING ALGORITHMS

PARTITIONING

4 8 17 18 19 22 27
SORTING ALGORITHMS

PARTITIONING

4 8 17 18 19 22 27
SORTING ALGORITHMS

PARTITIONING

4 8 17 18 19 22 27
SORTING ALGORITHMS

PARTITIONING

4 8 17 18 19 22 27
SORTING ALGORITHMS

PARTITIONING

4 8 17 18 19 22 27
SORTING ALGORITHMS

PARTITIONING

4 8 17 18 19 22 27
SORTING ALGORITHMS

QUICKSORT CONCEPT
1. Pick a pivot (simplest pivot is the first element of the array)

2. Put all items smaller than the pivot to its left

3. Mark the pivot as sorted

4. You will now have two subarrays, pick a pivot for each
subarray and determine their array sizes

5. Recursion!
SORTING ALGORITHMS

QUICKSORT BASIC PSEUDOCODE

function quicksort(array)
less, equal, greater := three empty arrays
if length(array) > 1
pivot := select any element of array
for each x in array
if x < pivot then add x to less
if x = pivot then add x to equal
if x > pivot then add x to greater
quicksort(less)
quicksort(greater)
array := concatenate(less, equal, greater)
SORTING ALGORITHMS

QUICKSORT IMPROVED PSEUDOCODE

function quicksort(array)
if length(array) > 1
pivot := select any element of array
left := first index of array
right := last index of array
while left ≤ right
while array[left] < pivot
left := left + 1
while array[right] > pivot
right := right - 1
if left ≤ right
swap array[left] with array[right]
left := left + 1
right := right - 1
quicksort(array from first index to right)
quicksort(array from left to last index)
SORTING ALGORITHMS

QUICKSORT IN C++
See quick_sort.cpp
SORTING ALGORITHMS

TIME COMPLEXITY ANALYSIS OF BASIC QUICKSORT

‣ If the array is random, the time complexity is Θ(N*log(N))
because of a similar divide and conquer approach to
merge sort


‣ However, if the array is sorted, and you choose the first

element of the pivot always, quicksort will run in O(N2)
time!


‣ The pivot is crucial in improving the time complexity of

quicksort
SORTING ALGORITHMS

SOME NOTES ON QUICKSORT’S TIME COMPLEXITY

‣ It runs in amortized Θ(N*log(N)) time

‣ Selecting a random pivot can significantly improve your chances

of getting Θ(N*log(N)) time


‣ A common implementation is selecting the middle of the list as

your pivot, but what if you picked the highest or lowest element
as the initial pivot?

‣ The divide and conquer recursion of quicksort is not as

guaranteed as the divide and conquer used by merge sort
SORTING ALGORITHMS

SPACE COMPLEXITY OF QUICKSORT

‣ If you create a copy array for each recursion, it will have the
same space complexity as merge sort


‣ However, there’s an "easy" way to not have to do that using

multiplex indices (see quick_sort.cpp) and have O(1)
complexity because you will only need normal variables to
create the indices
 SORTING ALGORITHMS

SUMMARY
Sorting Average Time Worst Time Space
Algorithm Complexity Complexity Complexity
Bubble Sort Θ(N2) or Θ(N) O(N2) O(1)

Selection Sort Θ(N2) O(N2) O(1)

Merge Sort Θ(N*log(N)) O(N*log(N)) O(N*log(N))

Ω(N) if array is
Insertion Sort O(N2) O(1)
almost sorted
Θ(N*log(N)) O(N*log(N))
Quick Sort O(N2)
amortized or O(1)
SEE YOU
THURSDAY :D