ENGR24524D ALGORITHMS AND DATA

STRUCTURES

,
Nazrul Khan PhD, PEng.

1
 Sorting and Ordering

Sorting: Definition
- Taking a list of objects which could be stored in a linear order (a 0, a1,
….an-1)
- Returning and reordering (a`0, a`1,…a`n-1)
- Whereby, a`0 ≤ a`1… ≤ a`n-1)
- The objective is to rearrange the items such that their keys are in
ascending order (low to high)

The conversion of an Abstract List into an abstract Sorted List.

- Usually, we will sort a number of records containing a
number of fields based on a key
 Sorting and Ordering

Sorting: Problem
 Sorting and Ordering

Sorting: Applications (Examples)

 Sorting and Ordering

Elementary Sorts: Selection & Insertion Sort:

Selection Sort:
- A sorting algorithm that rearranges the elements of a collection
- To be stored in a sorted manner
Example:

Step 1: Find the smallest element (5) and store in a[0] through
swapping with a[3]
Step 2: Find the minimum of the remaining entries, a[1] …a[4]
 Sorting and Ordering

Selection Sort:
Step 3: Next minimum (9) is already in the correct place. Nothing to
swap. Simply extend the sorted area by one to the right.

Step 4: Repeat the process. The minimum value of the unsorted region
is 11, which needs to be swapped with the first value of the unsorted
region, 17
 Sorting and Ordering

Selection Sort:
Step 5: Now, the unsorted region is only two elements long. The
minimum value is 12, and we swap it with the first value 17.

Step 6: Now, we have a unprocessed region of length 1. The region of

length 1 is always sorted. We are done!
 Sorting and Ordering

Selection Sort: Animations

Algorithm: scans from left to right (select one location(index) at a time).
Invariants:
 Sorting and Ordering

Selection Sort: Animations

 Sorting and Ordering

Selection Sort: animations

 Sorting and Ordering

Selection Sort: Implementation tips

- Pick the smallest element
- Swap with the first one (element at index 0)
- Pick the smallest element of the remaining ones
- Swap it with the next one (element at index 1)
- And so on…
 Sorting and Ordering

Selection Sort: Code Snippet

public void sort(int[ ] a)
{//Sort a[ ] into ascending order (minimum to maximum)
int N = a.length;
for(int i = 0; i < N; i++)
{//Exchange a[ i ] with smallest entry in a [i+1....N)
int min = i; //index of a minimum entry
for(int j = i+1; j < N; j++)
{
if(less(a[j], a[min]))
min = j;
}
exch (a,i,min);
}
}
 Sorting and Ordering

Selection Sort in action

 Sorting and Ordering

Selection Sort: Rules of Game

Certification: Does the sort implementation always put the
array in order?
Running Time: Algorithm performance
 Number of basic operations
 Compares, exchanges, accessed etc.,
Extra Memory: Amount of extra memory used by the
sorting algorithm.
Types of data: How effective the algorithm for all data
types.
 Sorting and Ordering

Insertion Sort:

Consider the following Observations:

- A list with one element is sorted
- In general, if we have a sorted list of k items, we can
insert a new item to create a sorted list of size k + 1
 Sorting and Ordering

Insertion Sort:
For example, consider this sorted array containing of eight sorted
entries

Suppose we want to insert 14 into this array leaving the resulting array
sorted
 Sorting and Ordering

Insertion Sort:
Starting at the back, if the number is greater than 14, copy it to the right
Once an entry less than 14 is found, insert 14 into the resulting
vacancy
 Sorting and Ordering

Insertion Sort: Algorithm

For any unsorted list:

- Treat the first element as a sorted list of size 1

Then, given a sorted list of size k – 1

- Insert the kth item in the unsorted list into it into the sorted
list
- The sorted list is now of size k+1
 Sorting and Ordering

Insertion Sort: Algorithm

In a computer implementation:

- We need to make space to insert the current item by

moving larger items one position to the right, before
inserting the current item into the vacated position
- The items to the left of the current index are in sorted
order during the sort but not in their final position
- The array is, however, fully sorted when the index
reaches the right end.
 Insertion sort
20

Sorting and Ordering

Five sorting techniques:
• Insertion

• Exchange

• Selection

• Merging

• Distribution

Clearly insertion falls into the first category

 Insertion sort
21

Sorting and Ordering

Insertion Sort: Code Snippet (partially sorted array)

for ( int j = k; j > 0; --j ) {

if ( array[j - 1] > array[j] ) {
swap( array[j - 1], array[j] );
} else {
// As soon as we don't need to swap, the (k + 1)st
// is in the correct location
break;
}
}
 Insertion sort
22

Sorting and Ordering

Insertion Sort: Code Snippet (unsorted array)

for(int k = 1; k < n; k++) {
for ( int j = k; j > 0; --j ) {
if ( array[j - 1] > array[j] ) {
swap( array[j - 1], array[j] );
} else {
// As soon as we don't need to swap, the (k + 1)st
// is in the correct location
break;
}
}
}
 Sorting and Ordering

Insertion Sort: Code Snippet

public void sort(int[ ] a)

{//Sort a[ ] into ascending order (minimum to maximum)
int N = a.length;
for(int i = 1; i < N; i++)
{//Insert a[ i ] among a[i-1], a[i-2], a[i-3], ……
for(int j = i; j > 0 && less(a[ j ], a[j-1]); j--)
{
exch (a, j, j-1);
}
}
}
 Sorting and Ordering

Insertion Sort in action

 Sorting and Ordering

Insertion Sort: Characteristics

- The algorithm is easy to implement

- Even in the worst case, the algorithm is fast for small problems
- A random list will have d = O(N2 ) inversions
- Approximately, 10 instructions per inversion
- Uses ~ N2 /4 compares and ~ N2 /4 exchanges
- The number of exchanges used by insertion sort is equal to the
number of inversions in the array
- The number of compares is at least equal to the number of
inversions .
 Sorting and Ordering

Insertion Sort: Partially Sorted Array

- An Array where each entry is not far from its final position
- A small array appended to a larger sorted array
- An array with only a few entries that are not in place

Insertion sort is an efficient method for such arrays; Selection sort is

not. Where number of inversions is low, insertion sort is the choice.
 Sorting and Ordering

Bubble Sort: Background

- Uses an opposite strategy from insertion sort
- Starting at the front, traverse the array, find the largest
item, and move (bubble) it to the top
- With each subsequent iteration, find the next largest item
and bubble it up towards the top of the array
- Bubble sort is a simple algorithm with:
 A memorable name, and
 A simple idea
 Significantly worse than insertion sort
 Sorting and Ordering

Bubble Sort: Implementation

- Starting with the first item, assume, that is the largest

- Compare it with the second item:
 If the first is larger, swap the two
 Otherwise, assume that the second item is the largest

Continue up the array, either swapping or redefining the

largest item
 Sorting and Ordering

Bubble Sort: Implementation

- After one pass, the largest item must be the last in the
list
- Start at the front again
- The second pass will bring the second largest element
into the second last position
- Repeat n – 1 times, after which, all entries will be in
place
 Sorting and Ordering

Bubble Sort: Algorithm

The default algorithm:

for ( int i = n - 1; i > 0; --i ) {

for ( int j = 0; j < i; ++j ) {
if ( array[j] > array[j + 1] ) {
swap( array[j], array[j + 1] );
}
}
}
 Sorting and Ordering

Bubble Sort: Run Time

Here we have two nested loops, and therefore calculating the run
time is straight-forward:

n 1
n  n  1 n  n  1

k 1
 n  k   n  n  1 
2

2
  ( n 2
)
 Sorting and Ordering

Bubble Sort: Example

Consider the unsorted array to
the right

We start with the element in the

first location, and move forward:
– if the current and next items
are in order, continue with the
next item, otherwise
– swap the two entries
 Sorting and Ordering

Bubble Sort: Example

After one loop, the largest

element is in the last location
– Repeat the procedure
 Sorting and Ordering

Bubble Sort: Example

Now the two largest elements

are at the end
– Repeat again
 Sorting and Ordering

Bubble Sort: Example

With this loop, 5 and 7 are

swapped
 Sorting and Ordering

Bubble Sort: Example

Finally, we swap the last two

entries to order them
– At this point, we have a
sorted array
 Sorting and Ordering
Bubble Sort: Improvement
We could avoid so many swaps...
for ( int i = n - 1; i > 0; --i ) {
Type max = array[0]; // assume a[0] is the max

for ( int j = 1; j <= i; ++j ) {

if ( array[j] < max ) {
array[j - 1] = array[j]; // move
} else {
array[j – 1] = max; // store the old max
max = array[j]; // get the new max
}
}

array[i] = max; // store the max

}
 Sorting and Ordering

Bubble Sort: Summary

it sounds as if it is as good as insertion sort

– it has the same asymptotic behaviour
– in practice, however, it is significantly worse
– it is also much more difficult to code...
 Sorting and Ordering

Heap Sort: Background

- A comparison based sorting algorithm
- Improved selection sort
- Inserting n objects into an a min-heap/max-heap and
then taking out n objects will results in them coming out
in order

Strategy:
- Given an unsorted list with n objects, place them into a
heap, and take them out.
 Sorting and Ordering

Heap Sort: In-place Implementation (max-heap)

- A heap where the maximum element is at the top of the
heap and the next to be popped.
 Sorting and Ordering

Heap Sort: In-place Heapification

Consider the unsorted array:

This array represents the following complete tree:

This is neither a min-heap, max-heap, or binary search

tree
 Sorting and Ordering

Heap Sort: In-place Heapification

Index (k) starts at 0!

For parent (at index k): index of: Left child= 2*k + 1, Right child=
2*k + 2
For a Child (at index k): index of Parent: (k + 1)/2 - 1
 Sorting and Ordering

Heap Sort: In-place Heapification

Can we convert this complete tree into max-heap?

- Operation must be done in-place
 Sorting and Ordering

Heap Sort: In-place Heapification

Two strategies:
– Assume 46 is a max-heap and keep inserting the next element into the
existing heap (similar to the strategy for insertion sort)
– Start from the back: note that all leaf nodes are already max heaps,
and then make corrections so that previous nodes also form max heaps
 Sorting and Ordering

Heap Sort: In-place Heapification (Example)

Let’s work bottom-up: each leaf node is a max heap on its own
 Sorting and Ordering

Heap Sort: In-place Heapification

Starting at the back, we note that all leaf nodes are trivial heaps
Also, the subtree with 87 as the root is a max-heap
 Sorting and Ordering

Heap Sort: In-place Heapification

The subtree with 23 is not a max-heap, but swapping it with 55

creates a max-heap
This process is termed percolating down
 Sorting and Ordering

Heap Sort: In-place Heapification

The subtree with 3 as the root is not max-heap, but we can swap 3
and the maximum of its children: 86
 Sorting and Ordering

Heap Sort: In-place Heapification

Starting with the next higher level, the subtree with root 48 can be
turned into a max-heap by swapping 48 and 99
 Sorting and Ordering

Heap Sort: In-place Heapification

Similarly, swapping 61 and 95 creates a max-heap of the next subtree
 Sorting and Ordering

Heap Sort: In-place Heapification

As does swapping 35 and 92
 Sorting and Ordering

Heap Sort: In-place Heapification

The subtree with root 24 may be converted into a max-heap by first
swapping 24 and 86 and then swapping 24 and 28
 Sorting and Ordering

Heap Sort: In-place Heapification

The right-most subtree of the next higher level may be turned into a
max-heap by swapping 77 and 99
 Sorting and Ordering

Heap Sort: In-place Heapification

However, to turn the next subtree into a max-heap requires that 13
be percolated down to a leaf node
 Sorting and Ordering

Heap Sort: In-place Heapification

The root (81) needs to be percolated down by two levels
 Sorting and Ordering

Heap Sort: In-place Heapification

The final product is a max-heap
 Sorting and Ordering

Heap Sort: Run-time Analysis of Heapify

Considering a perfect tree of height h:
– The maximum number of swaps which a second-lowest level would
experience is 1, the next higher level, 2, and so on
 Sorting and Ordering

Heap Sort: Run-time Analysis of Heapify

At depth k, there are 2k nodes and in the worst case, all of these
nodes would have to percolated down h – k levels
– In the worst case, this would requiring a total of 2k(h – k) swaps

Writing this sum mathematically, we get:

  
2 k h  k   2 h 1  1  ( h  1)
k 0
 Sorting and Ordering

Heap Sort: Example

Let us look at this example: we must convert the unordered array
with n = 10 elements into a max-heap

None of the leaf nodes need to

be percolated down, and the first
non-leaf node is in position n/2

Thus we start with position 10/2 = 5

 Sorting and Ordering

Heap Sort: Example

We compare 3 with its child and swap them
 Sorting and Ordering

Heap Sort: Example

We compare 17 with its two children and swap it with the maximum
child (70)
 Sorting and Ordering

Heap Sort: Example

We compare 28 with its two children, 63 and 34, and swap it with
the largest child
 Sorting and Ordering

Heap Sort: Example

We compare 52 with its children, swap it with the largest
– Recursing, no further swaps are needed
 Sorting and Ordering

Heap Sort: Example

Finally, we swap the root with its largest child, and recurse,
swapping 46 again with 81, and then again with 70
 Sorting and Ordering

Heap Sort: Example

We have now converted the unsorted array

into a max-heap:
 Sorting and Ordering

Heap Sort: Example

Suppose we pop the maximum element of this heap

This leaves a gap at the back of the array:

 Sorting and Ordering

Heap Sort: Example

This is the last entry in the array, so why not fill it with the largest
element?

Repeat this process: pop the maximum element, and then insert it
at the end of the array:
 Sorting and Ordering

Heap Sort: Example

Repeat this process
– Pop and append 70

– Pop and append 63

 Sorting and Ordering

Heap Sort: Example

We have the 4 largest elements in order
– Pop and append 52

– Pop and append 46

 Sorting and Ordering

Heap Sort: Example

Continuing...
– Pop and append 34

– Pop and append 28

 Sorting and Ordering

Heap Sort: Example

Finally, we can pop 17, insert it into the 2nd location, and the
resulting array is sorted
 Sorting and Ordering

Heap Sort: Example

Sort the following 12 entries using heap sort

34, 15, 65, 59, 79, 42, 40, 80, 50, 61, 23, 46
 Sorting and Ordering

Heap Sort: Run-Time

Heapification runs in Q(n)

Popping n items from a heap of size n, as we saw, runs in Q(n ln(n))

time
– We are only making one additional copy into the blank left at the end of
the array

Therefore, the total algorithm will run in Q(n ln(n)) time

 Sorting and Ordering

Heap Sort: Run Time

Taking an object out of a heap with n items requires O(ln(n)) time
Therefore, taking n objects out requires

n
 n


k 1
ln( k )  ln 



k 1
k   ln n!



Recall that ln(a) + ln(b) = ln(ab)

 Sorting and Ordering

Quick Sort: Back ground

- We will now look at a recursive algorithm which may be done almost in

place but faster than heap sort.
 Use an object in the array (a pivot) to divide
 Average case: Q(n ln(n)) time and Q(ln(n)) memory
 Worst case: Q(n2) time and Q(n) memory

– Choose an object in the array and partition the remaining

objects into two groups relative to the chosen entry.
 Sorting and Ordering

Quick Sort: Example

For example, given

we can select the middle entry, 44, and sort the remaining entries
into two groups, those less than 44 and those greater than 44:

Notice that 44 is now in the correct location if the list was sorted
– Proceed by applying the algorithm to the first six and last eight entries
 Sorting and Ordering

Quick Sort: Worst- case Scenario

Suppose we choose the first element as our pivot and we try
ordering a sorted list:

Using 2, we partition into

We still have to sort a list of size n – 1

The run time is T(n) = T(n – 1) + Q(n) = Q(n2)

– Thus, the run time change from n ln(n) to n2
 Sorting and Ordering

Quick Sort: Implementation

- Uses divide-and-conquer strategy
- An element (pivot) is first selected
- The array/list is partitioned into two subsets
- The subsets may or may not be equal in size
- In each subset:
 All the elements less than the pivot are towards the left of the
pivot
 All the elements greater than the pivot are towards the right of
the pivot
- Quick sort recursively sorts the two sub-lists.
 Sorting and Ordering

Quick Sort: Partitioning

- Partitioning is the key process
- Choose a value x, called pivot such that
 All the values less than x are before x
 All the values greater than x are after x
- A pivot value can be any of the following:
 First element in the array/list
 Last element in the array/list
 Mid element in the array/list
 Random element in the array/list
- The pivot value is then placed at its proper position in the array by
partitioning the array.
 Sorting and Ordering

Quick Sort: Partitioning Process

 Sorting and Ordering

Quick Sort: Partitioning

In the above diagram:

- Partitioning the array by repeatedly selecting the last element as the

pivot
- At each level, the array is partitioned into two subsets by placing the
pivot at its correct position.

The Partitioning Process:

5 40 15 10 32 5 7
 Sorting and Ordering

Quick Sort: Partitioning

5 40 15 10 32 5 7

5 7 8 40 15 10 32

8 15 10 32 40

8 10 15
 Sorting and Ordering

Quick Sort: Algorithm

The general algorithm(Recursive)

quicksort (Arr, low, high)
begin
declare array Arr [N] to be sorted
low = 1st element; high = last element; pivot
if(low < high)
begin
pivot = partition (Arr, low, high);
quicksort(Arr,low,pivot-1)
quicksort(Arr,pivot+1,high)
end
End
 Sorting and Ordering
Quick Sort: Illustration (Last element as the pivot)
 Sorting and Ordering

Quick Sort: Illustration

- Low points to first element. High points to last(pivot)
- When low element becomes greater than the pivot and high element
is lesser than the pivot element:
 Exchange the elements ( low <-> high)
 Low/high advances by 1 position
- The above process is repeated until both the pointers cross each
other. Once they cross:
 The pivot element positioned correctly
 The array is partitioned
- We can sort each sub-array independently recursively applying a
quick sort algorithm to each of the sub-array.
 Sorting and Ordering

Quick Sort: Summary

– On average faster than heap sort or merge sort

– Uses a pivot to partition the objects
– Using the median of three pivots is a reasonably
means of finding the pivot
– Average run time of Q(n ln(n)) and Q(ln(n)) memory
– Worst case run time of Q(n2) and Q(n) memory
 Acknowledgements
The above slides are adopted from:

1. Douglas Wilhelm Harder@uwaterloo

2. Algs4.cs@princeton
3. JavaCocepts@CayHorstmann