Chapter 2 Getting Started

1

Insertion Sort
Insertion-Sort(A)
sort an array A[1..n] containing a sequence of length n (length[A])
input array A contains the sorted output sequence when finished

input numbers are sorted in place
the numbers are arranged within the array A, with at most a constant number of them stored outside the array at any time
2

(i.e., n)

3

Pseudo Code Notations
Liberal use of English
employs whatever expressive method is most clear and concise to specify a given algorithm
algorithms are expressed to humans, not to computers

Use of indentation for block structure Omission of error handling and other such things needed in real programs
not concerned with issues of software engineering in order to convey the essence of the algorithm more concisely
4

Loop Invariants
Technique for proofs of the correctness of algorithms Conditions and relationships that are satisfied by the variables and data structures at the end of each iteration of the loop Established by induction on the number of passes through the loop
similar to mathematical induction
to prove a property holds, prove a base case and an inductive step
5

Properties of Loop Invariants
Initialization
it is true prior to the first iteration of the loop

Maintenance
if it is true before an iteration of the loop, it remains true before the next iteration

Termination
when the loop terminates, the invariant gives us a useful property that helps show that the algorithm is correct

6

Loop Invariants of Insertion Sort
Initialization
show that the loop invariants holds before the first loop iteration
j = 2 ⇒ A[1 .. j - 1] = A[1] is in sorted order

Maintenance
show that each iteration maintains the loop invariant
at the start of each iteration of the ”outer” for loop, the subarray A[1 .. j - 1] consists of the elements originally in A[1 .. j - 1] but in sorted order

Termination
examine what happens when the loop terminates
j = n + 1 ⇒ A[1 .. j - 1] = A[1 .. n]
7

Analyzing Algorithms
To predict the resources that the algorithm requires
resource examples
memory, communication bandwidth, computer hardware, computational time (measured most often)

Computational model
random-access machine (RAM) model
instructions are executed one after another
each instruction takes a constant amount of time

no concurrent operations data types are integer and floating point
the size of each word of data does not grow arbitrarily

no caches or virtual memory
no memory-hierarchy effects
8

Primitive Instructions
Basic computations performed by an algorithm
exact definition is not important assumed to take a constant amount of time in the RAM model

Examples
arithmetic: add, subtract, multiply, divide, remainder, floor, ceiling), shift left/shift right data movement: load, store, copy control: conditional/unconditional branch, subroutine call and return
9

Running Time
The number of primitive operations (steps) executed
steps are machine-independent each line of pseudo code requires a constant amount of time.
one line may take a different amount of time than another, but each execution of line i takes the same amount of time ci

10

Running Time
Depends on
input size
the time generally grows with the size of the input e.g., 6 elements vs. 6000 elements

input itself
may take different amounts of time on two inputs of the same size e.g., sorted vs. reversely sorted

11

Running Time Analysis
Worst case
maximum time on any input of size n usually used

Average case
average time over all inputs of size n occasionally used

Best case
minimum time on any input of size n hardly used

12

execute tj times for iteration j

Running Time: depends on the values of tj

• tj : the number of times the while loop test in line 5 is executed for the value of j

13

Best Case of Insertion Sort
Array already sorted
always A[i] ≤ key upon the first time the while loop test is run (when i = j - 1) all tj are 1

∑ t j = ∑1 = n − 1
j =2
n

n

n

j =2

∑t
j =2

j

− 1 = ∑1 − 1 = 0
j =2

n

14

Best Case Running Time

=

an + b for constants a and b (that depend on the statement costs ci ) ⇒ T(n) is a linear function of n

15

Worst Case of Insertion Sort
Array in reverse sorted order (i.e., decreasing order)
always A[i] > key in while loop test compared with j - 1 elements one additional test after the j - 1 tests ⇒ tj = j

∑t j = ∑ j =
j =2 n j =2 n j =2 j =2

n

n

n(n + 1) −1 2 n(n − 1) 2
16

∑t j −1 = ∑ j −1 =

Worst Case Running Time
∑j=
2 n

n( n + 1) −1 2

∑ ( j − 1) =
2

n

n( n − 1) 2

= an2+bn+c for constants a, b and c (that depend on the statement costs ci ) ⇒ T(n) is a quadratic function of n

17

Worst-Case Analysis
Worst-case running time is usually preferred
the longest running time for any input of size n

Reasons
the worst-case running time is an upper bound on the running time for any input for some algorithms, the worst case occurs fairly often
e.g., the worst case of searching often occurs when
the item being searched for is not present searches for absent items may be frequent

the average case is often roughly as bad as the worst case
18

Order of Growth
Use abstraction to ease analysis and focus on the important features
consider only the leading term of formula and drop lower-order terms
e.g., an2 instead of an2 + bn + c

ignore the constant coefficient in the leading term
e.g., n2 instead of an2

use Θ-notation as the running time
e.g., Θ(n2) ⇒ the worst-case running time T (n) grows like n2, but it does not equal n2
19

Algorithm Efficiency
One algorithm is considered to be more efficient than another if its worst case running time has a smaller order of growth
the evaluation may be in error for small inputs
due to constant factors and lower order terms

the evaluation will hold for large enough inputs
e.g., Θ(n2) algorithm will run more quickly in the worst case than a Θ(n3) algorithm

20

Designing Algorithms
Incremental approach
e.g., insertion sort, having sorted the subarray A[1 .. j - 1], inserts the single element A[j] into its proper place, yielding the sorted subarray A[1 .. j]

Divide and conquer approach
e.g., merge sort recursively sorted two sequences and merge the two sorted sequences into a bigger sorted sequence

21

Divide and Conquer Paradigm
Divide the problem into a number of subproblems
the subproblems are similar to the original problem but smaller in size

Conquer the subproblems by solving them recursively
if the subproblem sizes are small enough, however, just solve the problems in a straightforward manner

Combine the solutions to the subproblems into the solution to the original problem
22

Merge Sort Approach
Divide
divide the n-element sequence to be sorted into two subsequences of n/2 elements each

Conquer
sort the two subsequences recursively using merge sort

Combine
merge the two subsequences to produce the sorted answer

23

Merge Sort Procedure
State each subproblem as sorting a subarray A[p . . r]
initially, p = 1 and r = n the values of p and r change as recurring through subproblems

To sort A[p . . r]
divide by splitting into two subarrays A[p .. q] and A[q + 1 .. r], where q is the halfway point of A[p .. r] conquer by recursively sorting the two subarrays A[p .. q] and A[q + 1 .. r]
the recursion bottoms out when the subarray has just 1 element, which is trivially sorted

combine by merging the two sorted subarrays A[p .. q] and A[q + 1 .. r] to produce a single sorted subarray A[p .. r]
procedure MERGE(A, p, q, r)
24

Merge Sort Pseudocode
Initial call: MERGE-SORT(A, 1, n)
A = 〈A[1], A[2], …, A[n]〉

25

Bottom View of Merge Sort
n=8 n = 11

26

Merging
Input: Array A and indices p, q, r such that
p≤q<r subarray A[p .. q] is sorted and subarray A[q + 1 .. r] is sorted
by the restrictions on p, q, r, neither subarray is empty

Output: the two subarrays are merged into a single sorted subarray in A[p .. r] Running time: Θ(n) time
n = r - p + 1 = the number of elements being merged

27

Θ(1) → Θ(1) → Θ(1) →

{ Θ(n ) {
Θ(n1)
2

Θ(n1 + n2) = Θ(n)

Θ(1) → Θ(1) → Θ(1) → Θ(1) →    Θ(n)    

}

sentinel value

28

Example
A call of MERGE(A, 9, 12, 16)

29

Example(cont.)

30

Loop Invariant for Merge Sort
At the start of each iteration of the for loop of lines 12-17, the subarray A[p .. k-1] contains the k - p smallest elements of L[1 .. n1+1] and R[1 .. n2+1], in sorted order L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A

31

Loop Invariants of Merge Sort
Initialization
show that the loop invariants holds prior to the first iteration of for loop
k = p ⇒ A[p .. k - 1] is empty, k - p = 0 elements i=j=1

Maintenance
show that each iteration maintains the loop invariant
case L[i] ≤ R[i] case R[i] ≤ L[i]

Termination
examine what happens when the loop terminates
k = r + 1 ⇒ A[p .. j - 1] = A[p .. r]
32

Analyzing Divide-andConquer Algorithms
Use a recurrence equation (more commonly, a recurrence)
describes the overall running time on a problem of size n in terms of the running time on smaller inputs use mathematical tools later to solve the recurrence and provide bounds on the performance of the algorithm a recurrence for the running time of a divide-andconquer algorithm is based on the three steps of the basic paradigm
33

Running Time of Divideand-Conquer Algorithms
Let T(n) = running time on a problem of size n
if the problem size is small enough (say, n ≤ c for some constant c), we have a base case
the straightforward solution takes constant time: Θ(1)

otherwise, suppose that we divide into a subproblems, each 1/b the size of the original (in merge sort, a = b = 2)
there are a subproblems to solve, each of size n/b ⇒ each subproblem takes T(n/b) time to solve ⇒ solving a subproblems takes aT(n/b) time

let the time to divide a size-n problem be D(n) let the time to combine solutions be C(n)
34

Recurrence of Divide-andConquer Algorithms

aT(n/b) : the time to solve a subproblems D(n) : the time to divide a size-n problem D(n) C(n) : the time to combine solutions

35

Analyzing Merge Sort
Assume that n is a power of 2 for simplicity
each divide step yields two subproblems, both of size exactly n/2

When n = 1
the base case occurs and takes constant time ⇒ Θ(1)

When n > 1
divide: just compute q as the average of p and r ⇒ D(n) = Θ(1) conquer: recursively solve 2 subproblems, each of size n/2 ⇒ aT(n/b) = 2T(n/2) combine: MERGE on an n-element subarray takes Θ(n) time ⇒ C(n) = Θ(n)
36

Recurrence of Merge Sort
⇐ T(n) ⇐ Θ(1) ⇐ Θ(1) ⇐ D(n) ⇐ T(n/2) ⇐ T(n/2) ⇐ Θ(n) ⇐ C(n)

Recurrence of the running time

D(n) + C(n) = Θ(1) + Θ(n) = Θ(n)

37

Worst-Case Running Time of Merge Sort
By the master theorem in Chapter 4, it can be shown that this recurrence has the solution T(n) = Θ(n lg n) Merge sort is faster than insertion sort (Θ(n2) worst-case time)
on small inputs, insertion sort may be faster for large enough inputs, merge sort will always be faster

38

Solving the Merge-Sort Recurrence
Let c be a constant that describes the running time for the base case as well as the time per array element of the divide and combine steps Rewrite the recurrence as

Draw a recursion tree, which shows successive expansions of the recurrence

39

Recursion Tree

40

Recursion Tree (cont.)
lg n + 1 levels

each level has total cost cn

total cost of all levels cn (lg n + 1) = cn lg n + cn ⇒ Θ(n lg n)
41