Algorithm Course

Lecture- 4
Algorithms Classification &
Sorting-I
 No. of Steps Upper bound n large
Algorithm 1 g(n) = 3n
f(n) = 2n+3
O(n)
Algorithm 2 f(n) = 4n+8 g(n) =5n

Algorithm 3 g(n) =n2 O(n2)

f(n) = n +8
2

O(n3)

O(log n)

Algorithm .. “Asymptotic Analysis”

2
“Asymptotic Analysis”
Calculating Running Determine O-notation
Time f(n) (upper bounds)
f(n) a polynomial .. f(n) =Z 3n2 + 4n
• Use the pseudo code
• Ignore constants Special case:
• Consider Coding If is f(n) a polynomial of degree d,
implementation then f(n) is O(nd)

f(n) a function of n …. f(n) = n2 + Log n

Iterative
Ex: Do Loop
f(n) = n
f(n) a function of f(n)
Recurrence Solving Recurrence:
Ex: Merge Sort -Iteration -Substitution
f(n) = n + -Recursion trees -Master theorem
2f(n/2)
Classification of Algorithms

• By Function

• By Implementation

• By Design Paradigm
 Classification of Algorithms

Function Implementation Design Paradigm

Recursive vs
Sorting Iterative Divide & Conquer
Searching Determ. vs Dynamic
non-Determ. Programming
Selection
Logical vs
Arithmatic Procedural Greedy method
Serial vs Heuristic
Text Parallel
 Classification by Function
Sorting Algorithms (Bubble, Selection, Insertion,
Heap ...etc.)

Searching Algorithmsl (inear Binary SearchHashing Binary,

Search tree ….etc.)

Selection Algorithms

Arithmetic Algorithms

Text Algorithms
Classification by implementation
Recursive or iterative

A recursive algorithm: calls itself repeatedly until a certain

limit
Iterative algorithms: use repetitive constructs like loops.

Deterministic or non-deterministic

Deterministic algorithm: solve the problem with a predefined

process
Non-deterministic algorithm: must perform guesses of best
solution at each step through the use of heuristics.
Classification by implementation

Logical or procedural

Procedural algorithm: follow a certain procedure

Logical Algorithm: uses controlled deduction. Apply rules

Serial or parallel

Serial Algorithms: Based on executing one instruction of

an
algorithm at a time.
Parallel algorithms: take advantage of computer
architectures to process several instructions at once
Classification by design paradigm
Divide and conquer
Repeatedly reduces an instance of a problem to one or more
smaller instances of the same problem (usually recursively),
until the instances are small enough to solve easily. Sub-
problems are independent.
Divide-&-conquer is best suited for the case when no
“overlapping subproblems” are encountered.
 Classification by design paradigm
Dynamic programming

Having the optimal solution to a problem from optimal

solutions to subproblems. Subproblems are dependent and
overlaped; we don’t know where to partition the problem.

DP avoid recomputing solutions

that have already been computed.

Subproblem are solved only once

and their solutions are stored.
But this is at the cost of space
 Classification by design paradigm
The greedy method
Solves the sub-problems from top down. We first need to find
the greedy choice for a problem, then reduce the problem to a
smaller one. The solution is obtained when the whole problem
disappears.

Using graphs

Many problems can be modeled as problems on graphs. A

graph exploration algorithms are used. This category also
includes the search algorithms and backtracking.
Classification by design paradigm
Linear programming

The problem is expressed as a set of linear inequalities and

then an attempt is made to maximize or minimize the inputs
Probabilistic

Those that make some choices randomly.

Heuristic

Whose general purpose is not to find an optimal solution, but

an approximate solution where the time or resources to find
a perfect solution are not practical.
Sorting Algorithm
The Problem of Sorting
Sorting Algorithms classification

• Comparison based vs Counting based

• In-Place vs Not-In-Place algorithm
• Internal structure of the algorithm

• Data Structure of the algorithm

Comparison based vs Counting based
Comparison based sorting technique:

Bubble Sort
Selection Sort
Insertion
Sort Heap
Sort Quick
Sort Merge
Sort

Counting based sorting technique:

Radix Sort
Bucket Sort
In-Place vs Not-In-Place algorithm
In-place : In In-Place algorithm, no additional data structure
or array is required for sorting.
Bubble Sort
Selection Sort
Insertion
Sort Heap
Sort Quick
Sort

Not-In-Place : In Not-In-Place algorithm additional ,data

structure or array is required for sorting.
Radix Sort.
Bucket Sort.
Merge Sort.
 Internal structure of the algorithm
•Swap-based sorts begin conceptually with the entire list, and
exchange particular pairs of elements moving toward a more
sorted list.

•Merge-based sorts creates initial "naturally" or "unnaturally"

sorted sequences, and then add either one element (insertion
sort) or merge two already sorted sequences.

• Tree-based sorts store the data, at least conceptually, in a

binary tree; either based on heaps, or based on search trees.

•Other sorts which use additional key-value information, such

as radix or bucket sort.
Techniques of the Algorithm

• Sorting by Insertion insertion sort, shellsort

• Sorting by Exchange bubble sort, quicksort

• Sorting by Selection selection sort, heapsort

• Sorting by Merging merge

sort

• Sorting by Distribution radix

sort
Importance of Sorting

1. Computers spend more time sorting than anything else,

historically 25% on mainframes.

2.Sorting is the best studied problem in computer science,

with a variety of different algorithms known.

3.Many the interesting ideas can be taught in the context of

sorting, such as divide-and-conquer, randomized algorithms,
and lower bounds.
 Applications of Sorting
Searching

Binary search lets you test whether an item is in a dictionary

Closest pair

Given n numbers, find the pair which are closest to each

other. Once the numbers are sorted, the closest pair will be
next to each other in sorted order

Element Uniqueness

Given a set of n items, are they all unique or are there any
duplicates? Sort them and do a linear scan to check adjacent
pairs
Applications of Sorting
Mode

Given a set of n items, which element occurs the largest

number of times? Sort them and do a linear scan to measure
the length of all adjacent runs.

Median and Selection

What is the kth largest item in the set?

Once the keys are placed in sorted order in an array, the kth
largest can be determined by looking in the kth position of
the array.
Applications of Sorting
Convex hulls

Given n points in two dimensions, find the smallest area

polygon which contains them all.
Convex hulls are the most important
building block for more
sophisticated geometric algorithms.

Text Functions

Alphabetic is the sorting of text strings. There is a built-in sort

routine as a library function
The Problem of Sorting
Elementary Sorting Methods

(Selection, Insertion, Bubble)

• Easier to understand the basic mechanisms of
sorting.
• Good for small files.
• Good for well-structured files that are relatively
easy to sort, such as those almost sorted.
• Can be used to improve efficiency of more
powerful methods.
Selection Sort:
Given a list, take the current element and exchange it with the
smallest element on the right hand side of the current element.
Selection Sort

Pseudocode:

for i = 1:n
k = i
for j = i+1:n
if a[j] < a[k], k = j
swap a[i,k]
end
Selection Sort Analysis

• The number of comparisons is Θ(n2) in all cases.

• For each i from 1 to n-1, there is:

one exchange and n-i comparisons
A total of:
n-1 exchanges and
(n-1) + (n-2) + . . . + 2 + 1 = n(n-
1)/2 comparisons
 Example; Selection
Sort
{38, 27, 43, 3, 9, 82, 10}

1{38, 27, 43, 3, 9, 82, 10}, i: 0, min: 3, minValue: 3

2{ 3, 27, 43, 38, 9, 82, 10}, i: 1, min: 4, minValue: 9
3{
3, 9, 43, 38, 27, 82, 10}, i: 2, min: 6, minValue: 10
4{ 3, 9, 10, 38, 27, 82, 43}, i: 3, min: 4, minValue: 27
5{ 3, 9, 10, 27, 38, 82, 43}, i: 4, min: 4, minValue: 38
6{ 3, 9, 10, 27, 38, 82, 43}, i: 5, min: 6, minValue: 43
7{ 3, 9, 10, 27, 38, 43, 82}, i: 6
Insertion Sort
Insertion Sort
Insertion Sort Complexity Analysis

• BEST CASE is when array is already sorted.

• Best case complexity 🡪 O(n)

• WORST CASE is when the array is already reverse sorted

in decreasing order.
• Worst-Case complexity 🡪 O(n2)

• AVERAGE CASE assumes a random distribution of data.

On the average, 1/2 of all the inner loops are performed
• Average Case complexity 🡪 O(n2)
Example; Insertion
Sort
{38, 27, 43, 3, 9, 82, 10}

{38, 27, 43, 3, 9, 82, 10}

1{27, 38, 43, 3, 9, 82, 10}, i: 1
2
3{27, 38, 43, 3, 9, 82, 10}, i: 2
4{ 3, 27, 38, 43, 9, 82, 10}, i: 3
5{ 3, 9, 27, 38, 43, 82, 10}, i: 4
6
7{ 3, 9, 27, 38, 43, 82, 10}, i: 5
{ 3, 9, 10, 27, 38, 43, 82}, i: 6
Exchange (Bubble) Sort
• Based on repeatedly exchanging adjacent elements, if
necessary. When no exchanges are required, the file is
sorted.
• Search for adjacent pairs that are out of order
• Switch the out of order keys
• Repeat this n-1 times
• After the first iteration, the last key is guaranteed to be
the
largest

77 42 35 12 101 5
 "Bubbling Up" the Largest Element
Traverse a collection of elements
– Move from the front to the end
– “Bubble” the largest value to the end using pair-wise
comparisons and swapping

1 2 3 4 5 6

42
77 ap4277 35 12 101
Sw
5
 "Bubbling Up" the Largest
Element
Traverse a collection of elements
– Move from the front to the end
– “Bubble” the largest value to the end using pair-wise
comparisons and swapping

1 2 3 4 5 6

77 35 ap 35
42 77 12 101 5
Sw
 "Bubbling Up" the Largest
Element
Traverse a collection of elements
– Move from the front to the end
– “Bubble” the largest value to the end using pair-wise
comparisons and swapping

1 2 3 4 5 6

7712 ap 12
42 35 77 101 5
Sw
 "Bubbling Up" the Largest
Element
• Traverse a collection of elements
– Move from the front to the end
– “Bubble” the largest value to the end using pair-wise
comparisons and swapping

1 2 3 4 5 6

42 35 12 77 101 5

No need to swap
 "Bubbling Up" the Largest
Element
• Traverse a collection of elements
– Move from the front to the end
– “Bubble” the largest value to the end using pair-wise
comparisons and swapping

1 2 3 4 5 6

42 35 12 77 5 ap 101
5
101 Sw
 "Bubbling Up" the Largest Element

• Traverse a collection of elements

– Move from the front to the end
– “Bubble” the largest value to the end using pair-
wise comparisons and swapping

1 2 3 4 5 6

42 35 12 77 5 101

Largest value correctly placed

 Items of Interest

• Notice that only the largest value is correctly

placed
• All other values are still out of order
• So we need to repeat this process

1 2 3 4 5 6

42 35 12 77 5 101

Largest value correctly placed

 Repeat “Bubble Up” How Many Times?
• If we have N elements…

• And if each time we bubble an element, we place it

in its correct location…

• Then we repeat the “bubble up” process N – 1

times.

• This guarantees we’ll correctly place all N

elements.
 “Bubbling” All the Elements

1 2 3 4 5 6
42 35 12 77 5 101
1 2 3 4 5 6
35 12 42 5 77 101
1 2 3 4 5 6
N-1

12 35 5 42 77 101
1 2 3 4 5 6
12 5 35 42 77 101
1 2 3 4 5 6
5 12 35 42 77 101
Bubble Sort Pseudo Code

for i = 1 to n do
for j = n to i+1 do
If A[A] < A[j-1] then
Exchange A[j] ↔
A[j-1]

• Inner for loop executed O(n2) times

• Best- and worst-case times same O(n2)