Design and Analysis of Algorithms

Lecture 12-14

Dr. Farhana Jabeen

Dr. Farhana Jabeen 1 / 36

 Lecture
Outline
 Iterative sorting algorithms (comparison based)
 Selection Sort
 Bubble Sort
 Insertion Sort
 Recursive sorting algorithms (comparison based)
 Merge Sort
 Quick Sort
 Radix sort (non-comparison based)
 Properties of Sorting
 In-place sort, stable sort
 Comparison of sorting algorithms
 Note: we only consider sorting data in ascending order

2 / 36
2
 Why Study
Sorting?
 When an input is sorted, many problems become
easy (e.g. searching, min, max, k-th smallest)

 Sorting has a variety of interesting algorithmic

solutions that embody many ideas
 Comparison vs non-comparison based
 Iterative
 Recursive
 Divide-and-conquer
 Best/worst/average-case bounds
 Randomized algorithms

3 / 36
3
 Applications of
Sorting
 Uniqueness testing
 Deleting duplicates
 Prioritizing events
 Frequency counting
 Reconstructing the original order
 Set intersection/union
 Finding a target pair x, y such that x+y = z
 Efficient searching

4 / 36
4
 Bubble Sort: Idea
 Given an array of n items
1. Compare pair of adjacent items
2. Swap if the items are out of order
3. Repeat until the end of array
 The largest item will be at the last position
4. Reduce n by 1 and go to Step 1

 Analogy
 Large item is like “bubble” that floats to the end of the
array

5 / 36
1
Bubble sort

• Start – Unsorted

• Compare, swap (0, 1)

• Compare, swap (1, 2)

• Compare, no swap

• Compare, noswap
• Compare, swap (4, 5)

• 99 in position
Dr. Farhana Jabeen 6 / 36
Bubble sort

• Pass 2

• swap (0, 1)

• no swap

• no swap

• swap (3, 4)

• 21 in position

Dr. Farhana Jabeen 7 / 36

Bubble sort

• Pass 3
• no swap
• no swap

• swap (2, 3)

• 12 in position, Pass 4

• no swap

• swap (1, 2)

• 8 in position, Pass 5

• swap (1, 2)

• Pass 6 no swap Done

Dr. Farhana Jabeen 8 / 36
 Bubble Sort:
Implementation
void bubbleSort(int a[], int n)
{ for (int i = n-1; i >= 1; i--) Step 1:
{ Compare
for (int
if j(a[j-1]
= 1; j ><=a[j])
i; j++) { adjacent
swap(a[j], a[j-1]); pairs of
numbers
}
}
Step 2:
} Swap if the
items are out
of order

29 10 14 37 13
http://visualgo.net/sorting?create=29,10,14,37,
13&mode=Bubble

9 / 36
 Bubble Sort:
Implementation
void bubbleSort(int a[], int n) { for 3*n
(int i = n-1; i >= 1; i--) {
for (int j = 1; j <= i; j++) { i n
3*i
if (a[j-1] > a[j]) i=1
swap(a[j], a[j-1]);
}
}
in
}
i
i=1

29 10 14 37 13
http://visualgo.net/sorting?create=29,10,14,37,
13&mode=Bubble

10 / 36
 Bubble Sort: Analysis
 1 iteration of the inner loop (test and swap) requires
time bounded by a constant c
 Two nested loops
 Outer loop: exactly n iterations
 Inner loop:
 when i=0, (n−1) iterations
 when i=1, (n−2) iterations
 …
 when i=(n−1), 0 iterations
 Total number of iterations = 0+1+…+(n−1) = n(n−1)/2
 Total time = c n(n−1)/2 = O(n2)

11 / 36
Bubble Sort

Dr. Farhana Jabeen 12 / 36

Bubble Sort: Worst Case

Dr. Farhana Jabeen 13 / 36

Bubble Sort: (BEST Case)

Dr. Farhana Jabeen 14 / 36

Bubble Sort

• In any cases, (worse case, best case or average

case) to sort the list in ascending order the
number of comparisons between elements is the
same.
–Worse Case [8 7 6 3 1]
– Average Case [7 8 3 1 6]
– Best Case [1 3 6 7 8]
• All lists with 5 elements need 10 comparisons to
sort all the data.

Dr. Farhana Jabeen 15 / 36

Bubble Sort

• In the example given, it can be seen that the number of

comparison for worse case and best case is the same ‐
with 10 comparisons.
• The difference can be seen in the number of swapping
elements. Worse case has maximum number of
swapping: 10, while best case has no swapping since all
data is already in the right position.
• For best case, starting with pass one, there is no
exchange of data occur.
• From the example, it can be concluded that in any pass,
if there is no exchange of data occur, the list is already
sorted.
• The next pass shouldn't be continued and the sorting
process should stop.

Dr. Farhana Jabeen 16 / 36

Bubble Sort

• Number of Swaps
– inside a conditional ‐> #swaps data dependent !!
– Best Case 0, or O(1)
– Worst Case (n‐1)+(n‐2)+(n‐3) …+1 , or O(n2)
• Bubble Sort takes several passes to sort elements in an
array. Every pass need to do comparisons between
elements and exchange the data if the elements are not
in the right order.
• However the complexity of Bubble sort is the same for
best case and worse case.

Dr. Farhana Jabeen 17 / 36

 Bubble-Sort Running Time
cost times
BUBBLESORT(A, n) c1

(n+1)
for i  1 to n n

 (n  i  1)
c2

for j  0 to n-i
i 1

c3
n
if A[j] < A[j -1]  (n  i)
i 1
n

 (n  i )
c4

then exchange A[j]  A[j-1] i 1

Dr. Farhana Jabeen 18 / 36

 Bubble-Sort Running Time

n n n
T(n) = c1(n+1) + c2  (n  i  1)  c  (n  i)  c  (n  i)
i 1
3
i 1
4
i 1
n
= O(n) + (c2 + c3 + c4)  (n  i )
i 1
n n
n ( nn
 1) n 2
n
where  (n  i )  n   i  n 2   
i 1 i 1 i 1 2 2 2
Thus,T(n) = O(n2)
Dr. Farhana Jabeen 19 / 36
 Bubble Sort: Early
Termination
 Bubble Sort is inefficient with a O(n2) time
complexity
 However, it has an interesting property
 Given the following array, how many times will the
inner loop swap a pair of item?

3 6 11 25 39

 Idea
If we go through the inner loop with no swapping


 the array is sorted

 can stop early!

20 / 36
 Bubble Sort v2.0:
Implementation
void bubbleSort2(int a[], int n) {
for (int i = n-1; i >= 1; i--) { Assume the array
bool is_sorted = true; is sorted before
the inner loop
for (int j = 1; j <= i; j++) { if
(a[j-1] > a[j]) {
swap(a[j], a[j-1]); Any swapping will
is_sorted = false; invalidate the
assumption
}
} // end of inner loop if
(is_sorted) return; If the flag
} remains true
} after the inner
loop  sorted!

21 / 36
 Bubble Sort v2.0:
Analysis
 Worst-case
 Input is in descending order
 Running time remains the same: O(n2)

 Best-case
 Input is already in ascending order
 The algorithm returns after a single outer iteration
 Running time: O(n)

22 / 36
Selection Sort

• Idea:
– Find the smallest element in the array
– Exchange it with the element in the first position
– Find the second smallest element and exchange it
with the element in the second position
– Continue until the array is sorted
• Disadvantage:
– Running time depends only slightly on the amount of
order in the file

Dr. Farhana Jabeen 23 / 36

Selection Sort:
Idea
 Given an array of n items
1. Find the largest item x, in the range of [0…n−1]
2. Swap x with the (n−1)th item
3. Reduce n by 1 and go to Step 1

24 / 36
2
4
 Selection Sort:
Illustration
37 is the largest, swap it with
29 10 14 37 13 the last element, i.e. 13.
Q: How to find the largest?

29 10 14 13 37
x Unsorted items
13 10 14 29 37 Largest item for
x current iteration

x Sorted items
13 10 14 29 37

10 13 14 29 37 Sorted!

We can also find

the smallest and
put it the front
instead 25 / 36
2
5
 Selection Sort:
Implementation
void selectionSort(int a[], int n) { for
(int i = n-1; i >= 1; i--) {
int maxIdx = i; Step 1:
for (int j = 0; j < i; j++) if (a[j] >= Search for
a[maxIdx]) maximum
element
maxIdx = j;
// swap routine is in STL <algorithm>
swap(a[i], a[maxIdx]);
} Step 2:
} Swap
maximum
element
with the last
item i

26 / 36
2
6
Dr. Farhana Jabeen 27 / 36
Selection Sort

• Selection sort stops, when unsorted part

becomes empty.
• Selection sort makes n steps (n is number of
elements in array) of outer loop, before stop.
• Every step of outer loop requires finding
minimum in unsorted part.
• Summing up, n + (n - 1) + (n - 2) + ... + 1,
results in O(n2) number of comparisons.

Dr. Farhana Jabeen 28 / 36

Selection Sort

Alg.: SELECTION-SORT(A) 8 4 6 9 2 3 1
n ← length[A]
for j ← 0 to n - 1 for (j=0;j<=n-1,j++)
do small_pos ← j
for i ← j + 1 to n
do if A[i] < A[small_pos]
then small_pos ← i
exchange A[j] ↔ A[small_pos]

Dr. Farhana Jabeen 29 / 36

Selection Sort

• How many compares are done?

– n+(n-1)+(n-2)+...+1, or O(n2)
• How many swaps are done?
– Note swap is run even if already in position n,
or O(n)
• How much space?
– In-place algorithm (note the similarity)

Dr. Farhana Jabeen 30 / 36

 Insertion Sort
• Idea: like sorting a hand of playing cards
– Start with an empty left hand and the cards facing
down on the table
– Remove one card at a time from the table, and insert
it into the correct position in the left hand
• compare it with each of the cards already in the hand, from
right to left
– The cards held in the left hand are sorted
• these cards were originally the top cards of the pile on the
table

31
31 / 36
Insertion Sort:
Idea
 Similar to how most people arrange a hand of
poker cards
 Start with one card in your hand
 Pick the next card and insert it into its proper sorted
order
 Repeat previous step for all cards

1st card: 10♠ 10♠

2nd card: 5♠
150
♠
3rd card: K♠ 5♠ 10♠ K♠

… … … …

32 / 36
 Insertion Sort
• Idea: like sorting a hand of playing cards
– Start with an empty left hand and the cards facing
down on the table.
– Remove one card at a time from the table, and insert
it into the correct position in the left hand
• compare it with each of the cards already in the hand, from
right to left
– The cards held in the left hand are sorted
• these cards were originally the top cards of the pile on the
table

33
33 / 36
 Insertion Sort

To insert 12, we need to

make room for it by moving
first 36 and then 24.
6 10 24 36

12

34
34 / 36
 Insertion Sort

6 10 24 36

12

35
35 / 36
 Insertion Sort

6 10 24 3
6

12

36
36 / 36
 Insertion Sort:
Illustration
Start 40 13 20 8
x Sorted

x Unsorted
Iteration 1 13 40 20 8
Unsorted
x To be inserted

Iteration 2 13 20 40 8

Iteration 3 8 13 20 40

http://visualgo.net/sorting?create=40,13,20,8&mode=Insertion

37 / 36
 Insertion Sort
input array

5 2 4 6 1
3
at each iteration, the array is divided in two sub-arrays:

left sub-array right sub-array

sorted unsorted

38
38 / 36
Insertion Sort

39
39 / 36
 INSERTION-SORT
Alg.: INSERTION-SORT(A) 1 2 3 4 5 6 7 8

a1 a 2 a3 a4 a 5 a6 a7 a 8
for j ← 2 to n
do key ← A[ j ] key
Insert A[ j ] into the sorted sequence A[1 . . j -1]

i←j-1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i←i–1
A[i + 1] ← key
• Insertion sort – sorts the elements in place
40
40 / 36
 Loop Invariant for Insertion Sort
Alg.: INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
Insert A[ j ] into the sorted sequence A[1 . . j -1]

i←j-1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i←i–1
A[i + 1] ← key
Invariant: at the start of the for loop the elements in A[1 . . j-1]
are in sorted order 41
41 / 36
 Analysis of Insertion Sort
INSERTION-SORT(A) cost times
for j ← 2 to n c1 n
do key ← A[ j ] c2 n-1
Insert A[ j ] into the sorted sequence A[1 . . j -1] 0 n-1
i←j-1 c4 n-1

n
t
while i > 0 and A[i] > key c5 j 2 j


n
(t j  1)
do A[i + 1] ← A[i] c6 j 2


n
i←i–1 (t j  1)
c7 j 2

A[i + 1] ← key c8 n-1

tj: # of times the while statement is executed at iteration j

T (n)  c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6   t j  1  c7   t j  1  c8 (n  1)
n n n

j  242 j 2 j 2
42 / 36
 Best Case Analysis
• The array is already sorted “while i > 0 and A[i] > key”
– A[i] ≤ key upon the first time the while loop test is run
(when i = j -1)

– tj = 1

• T(n) = c1n + c2(n -1) + c4(n -1) + c5(n -1) + c8(n-1) =

(c1 + c2 + c4 + c5 + c8)n + (c2 + c4 + c5 + c8)

= an + b = (n)
T (n)  c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6   t j  1  c7   t j  1  c8 (n  1)
n n n

j 43
2 j 2 j 2
43 / 36
 Worst Case Analysis
• The array is in reverse sorted order “while i > 0 and A[i] > key”
– Always A[i] > key in while loop test
– Have to compare key with all elements to the left of the j-th
position  compare with j-1 elements  tj = j
n
n(n  1) n
n(n  1) n
n(n  1)
using j 1
j
2
  j 
j 2 2
 1   ( j  1) 
j 2 2
we have:

 n( n  1)  n( n  1) n( n  1)
T ( n )  c1n  c2 (n  1)  c4 ( n  1)  c5   1  c6  c7  c8 ( n  1)
 2  2 2

 an 2  bn  c a quadratic function of n

• T(n) = (n2) order of growth in n 2

T (n)  c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6   t j  1  c7   t j  1  c8 (n  1)
n n n

44
j 2 j 2 j 2
44 / 36
 Comparisons and Exchanges in
Insertion Sort
INSERTION-SORT(A) cost times
c1 n
for j ← 2 to n
c2 n-1
do key ← A[ j ]
0 n-1
Insert A[ j ] into the sorted sequence A[1 . . j -1]
 n2/2 comparisons c4 n-1
i←j-1

n
c5 j 2 j
t
while i > 0 and A[i] > key
c6 
n
j 2
(t j  1)
do A[i + 1] ← A[i]

n
n 2
/2 exchanges c7 j 2
(t j  1)
i←i–1
c8 n-1
A[i + 1] ← key 45
45 / 36
 Insertion Sort:
Implementation
void insertionSort(int a[], int n) {
for (int i = 1; i < n; i++) { next is the
item to be
int next = a[i];
inserted
int j;

for (j = i-1; j >= 0 && a[j] > next; j--)

a[j+1] = a[j];
Shift sorted
items to make
a[j+1] = next;
place for next
}
} Insert next to
the correct
location

29 10 14 37 13
http://visualgo.net/sorting?create=29,10,14,37,13&mode=Insertion

46 / 36
Insertion Sort

• Insertion sort is an example of an incremental algorithm;

– sorted array (or list) is built one entry at a time.
• It is much less efficient on large lists compared to algorithms such
as quick sort, heap sort or merge sort.
• Insertion sort provides several advantages:
• Simple implementation
• Efficient for (quite) small data sets
• Efficient for data sets that are already substantially sorted

Dr. Farhana Jabeen 47 / 36

 Insertion Sort - Summary
• Advantages
– Good running time for “almost sorted” arrays
(n)
• Disadvantages
 (n2) running time in worst and average case
  n2/2 comparisons and exchanges

48
48 / 36
 Proving Loop Invariants
• Proving loop invariants works like induction
• Initialization (base case):
– It is true prior to the first iteration of the loop
• Maintenance (inductive step):
– If it is true before an iteration of the loop, it remains true before
the next iteration
• Termination:
– When the loop terminates, the invariant gives us a useful
property that helps show that the algorithm is correct
– Stop the induction when the loop terminates

49
49 / 36
 Loop Invariant for Insertion Sort
• Initialization:
– Just before the first iteration, j
= 2:
the subarray A[1 . . j-1] =
A[1], (the element originally in
A[1]) – is sorted

50
50 / 36
 Loop Invariant for Insertion Sort
• Maintenance:
– the while inner loop moves A[j -1], A[j -2],
A[j -3], and so on, by one position to the right
until the proper position for key (which has the
value that started out in A[j]) is found
– At that point, the value of key is placed into this
position.

51
51 / 36
 Loop Invariant for Insertion Sort
• Termination:
– The outer for loop ends when j = n + 1  j-1 =
n
– Replace n with j-1 in the loop invariant:
• the subarray A[1 . . n] consists of the jelements
-1 j
originally in A[1 . . n], but in sorted order

Invariant: at the start of the for loop the elements in A[1 . . j-1]
are in sorted order

• The entire array is sorted!

52
52 / 36
Example

CS 477/677 - Lecture 5 53 / 36 53
 Loop Invariant for Insertion Sort
Alg.: INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
Insert A[ j ] into the sorted sequence A[1 . . j -1]

i←j-1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i←i–1
A[i + 1] ← key
Invariant: at the start of each iteration of the for loop, the
elements in A[1 . . j-1] are in sorted order 54
54 / 36