373

Lecture 1
By
Javed Siddique
 Algorithm and Data Structure
• What is an Algorithm?
• What is a Data Structure?
 Why we study algorithms?
• To make efficient programs and software.
• Make efficient use of scarce resources.
• Time, Space, Bandwidth etc.
• Do we need to study algorithms if we have
an infinitely fast machine with infinite
amount of memory?
 Classification of Problems
• There are problems that has no solution
(Halting Problem).
• There are problems which does not have
any polytime solution (Traveling salesman
problem). (Hard Problems)
• There are problems which can be solved in
polytime.
 Problem
• Suppose we have an O(n^3) algorithm. It
takes 0.47 seconds to solve a problem with
input size 100. How long will it take if input
size is 1000.
Algorithm Matters
 Models of Computation
a) Random Access Machines
b) Straight Line Programs and Circuits
c) Vector Machines
d) Turing Machines
e) Pointer Machines
f) Decision Trees
g) Machines That Make Random Choices
RAM Model
 RAM Assumptions
1) Each register holds an integer
2) Program can’t modify itself
3) Memory instructions involve simple
arithmetic
a) Addition, subtraction
b) Multiplication, division and control states
(got, if-then, etc.)
 Correctness of Insertion Sort
• We need to prove that insertion sort is correct.
• One way to do this is to use loop invariant.
• Loop invariant is a statement that remains true throughout
the algorithm.
• We use loop invariants to help us understand why an
algorithm is correct. We must show three things about a
loop invariant:
– Initialization: It is true prior to the first iteration of the loop.
– Maintenance: If it is true before an iteration of the loop, it
remains true before the next iteration.
– Termination: When the loop terminates, the invariant gives us a
useful property that helps show that the algorithm is correct.
 Insertion Sort
• INSERTION-SORT(A)
• 1 for j ← 2 to length[A]
• 2 do key ← A[j]
• 3 ▹ Insert A[j] into the sorted sequence A[1 j - 1].
• 4i←j-1
• 5 while i > 0 and A[i] > key
• 6 do A[i + 1] ← A[i]
• 7i←i-1
• 8 A[i + 1] ← key
 Loop Invariant Statement
• At the start of each iteration of the for loop
of lines 1-8, the subarray A[1 j - 1] consists
of the elements originally in A[1 j - 1] but
in sorted order.
 Proof of Correctness
• Initialization: We start by showing that the
loop invariant holds before the first loop
iteration, when j = 2.[1] The subarray A[1 j
- 1], therefore, consists of just the single
element A[1], which is in fact the original
element in A[1]. Moreover, this subarray is
sorted (trivially, of course), which shows
that the loop invariant holds prior to the first
iteration of the loop.
 Proof of Correctness
• Maintenance: Next, we tackle the second property:
showing that each iteration maintains the loop invariant.
Informally, the body of the outer for loop works by
moving A[ j - 1], A[ j - 2], A[ j - 3], and so on by one
position to the right until the proper position for A[ j] is
found (lines 4-7), at which point the value of A[j] is
inserted (line 8). A more formal treatment of the second
property would require us to state and show a loop
invariant for the "inner" while loop. At this point,
however, we prefer not to get bogged down in such
formalism, and so we rely on our informal analysis to
show that the second property holds for the outer loop.
 Proof of Correctness
• Termination: Finally, we examine what happens
when the loop terminates. For insertion sort, the
outer for loop ends when j exceeds n, i.e., when j
= n + 1. Substituting n + 1 for j in the wording of
loop invariant, we have that the subarray A[1,n]
consists of the elements originally in A[1 n], but
in sorted order. But the subarray A[1 n] is the
entire array! Hence, the entire array is sorted,
which means that the algorithm is correct.
Example
 Designing Algorithms
• There are many ways to design algorithms
– Incremental
– Divide and Conquer
– Dynamic
– Greedy
– Bruteforce
– Probabilistic
– Etc.
 Incremental Approach
• Insertion sort uses an incremental
approach:
• Having sorted the subarray A[1 j - 1], we
insert the single element A[j] into its proper
place, yielding the sorted subarray A[1 j].
 Divide and Conquer
• Usually recursive in structure.
• The divide-and-conquer paradigm involves three
steps at each level of the recursion:
– Divide the problem into a number of subproblems.
– Conquer the subproblems by solving them recursively.
If the subproblem sizes are small enough, however, just
solve the subproblems in a straightforward manner.
– Combine the solutions to the subproblems into the
solution for the original problem.
 Merge Sort
• The merge sort algorithm closely follows the
divide-and-conquer paradigm. Intuitively, it
operates as follows.
– Divide: Divide the n-element sequence to be sorted into
two subsequences of n/2 elements each.
– Conquer: Sort the two subsequences recursively using
merge sort.
– Combine: Merge the two sorted subsequences to
produce the sorted answer.
Merge Sort
 Merge Sort
• MERGE-SORT(A, p, r)
• 1 if p < r
• 2 then q ← ⌊(p + r)/2⌋
• 3 MERGE-SORT(A, p, q)
• 4 MERGE-SORT(A, q + 1, r)
• 5 MERGE(A, p, q, r)
Merge Sort
 Analyzing Merge Sort
• Recurrence Relation
 Merge Sort
• Divide: The divide step just computes the middle
of the subarray, which takes constant time. Thus,
D(n) = Θ(1).
• Conquer: We recursively solve two subproblems,
each of size n/2, which contributes 2T (n/2) to the
running time.
• Combine: We have already noted that the
MERGE procedure on an n-element subarray
takes time Θ(n), so C(n) = Θ(n).
Runtime Analysis