01 Iterative

Iterative Algorithms
& Loop Invariants

Actions vs Assertions Don't Redo Work
Actions vs Loop Invariants Designing an Algorithm
Proof Using Loop Invariants Finding Errors
Induction Fairy God Mother (Lake)
Logic Game Infinite Row Game
Code vs Math Assertions More of Input (Insertion and DFA)
Physics and Life More of Output (Selection, Blocks, König)
Home to School Problem Narrowing the Search Space
Recap of Proof of Meta Algorithm The Partition Problem
Code from LI Shrinking Instance (GCD) (Running Time)
Pointers More Multiplying
Views of Algorithms Data Structure (System) Invariants
Insertion Sort Bucket (Quick) Sort for Humans
Code from LI - Sorting Radix/Counting Sort
Selection Sort Lower Bound for Sort
Find Errors
Cake Cutting
Matrix Search
Jeff Edmonds
Lecture 1 York University COSC 31011
Verify vs Create
“The Rules taught should be

the ones our forefathers devised.
All else is dangerous”.
Adding meaning (semantics)
or intuition is bad.

2
Verify vs Create
I will teach you to factor.
Input: An integer x
Output: Its prime factors p1,p2,…,pr
Verify: p1p2…pr = x
Input: x = 12
Output: Factors 2,2,3
Verify: 223 = 12
Input: x = 7152988453
Output: Factors 182983,39091
Verify: 18298339091 = 7152988453
Easy!
No problem. I know how to multiply.

I can verify that 18298339091 = 7152988453. 3
Verify vs Create
Now the test.
Input: x = 23434551
What are the factors?
Aaaah!
You never taught me how to factor.
That is much much harder!.
4
Verify vs Create
If you give me a bunch of algorithms
and teach me how to verify that they work.
This does not necessarily teach me
how to create new algorithms.
Let’s try.
Aaaah!
You never taught me how to factor.
That is much much harder!.
5
Actions vs Assertions
Paradigm Shift
Is the black the form and the green the background?

Is the green the form and the black the background?
It is helpful to have different ways of looking at it.
6
An Algorithm viewed
as a sequence of
Actions: Landmarks: ie an Assertion
of the static
picture
of what is true
.. Straight at that moment
.. Right in the journey.
.. Straight
.. Left …..
I can get lost!!! 7

Max( A ) My boss wants me to write code.
“preCond: He gives me:
Input is array A[1..n]
of n values.” The contract is
i=1 preCond  postCond
m = A[1] code
loop Preconditions:
“loop-invariant: Any assumptions that must be true
m is max in {A[1]..A[i]}” about the input instance.
exit when (i=n)
m = max(m,A[i+1]) Postconditions:
i=i+1 Any statement of what must be true
endloop when the algorithm/program returns.
return(m)
“postCond: I give him:
return max in {A[1]..A[n]}” 8
“preCond: α β αβ
He gives me:
Input is array A[1..n] T T T?
of n values.” T F The contract is
F T
F F T T
m = A[1] code
loop
“loop-invariant: If the client gives us an input that
m is max in {A[1]..A[i]}” meets the precondition
exit when (i=n) and my program gives an output
m = max(m,A[i+1]) that meets the postcondition,
i=i+1 then everyone is happy.
endloop
return(m)
“postCond:
“preCond: α β αβ
He gives me:
Input is array A[1..n] T T T
of n values.” T F The contract is
F T T
?
F F T F F
m = A[1] code
loop
“loop-invariant: If the client gives us an input that
m is max in {A[1]..A[i]}” does NOT meet the precondition
exit when (i=n) and my program gives an output
m = max(m,A[i+1]) that does NOT meet the
i=i+1 postcondition,
endloop then my boss can’t be mad.
return(m) My code meets the contract.
“postCond:
“preCond: α β α β
He gives me:
Input is array A[1..n] T T T
of n values.” T F F The contract is
F T T
F F T T F
m = A[1] code
loop
The only way to break my contract is if
“loop-invariant:
the input that meets the precondition
m is max in {A[1]..A[i]}”
and my output does NOT meet the postcondition.
exit when (i=n)
m = max(m,A[i+1])
i=i+1
endloop
return(m)
“postCond:
Max( A )
“preCond:
I implored you to not worry
about the entire computation.
of n values.”
i=1
m = A[1]
loop
“loop-invariant:
exit when (i=n)
m = max(m,A[i+1])
i=i+1
endloop
return(m)
next
“postCond:
Max( A )
“preCond:
of n values.”
Trying to understand
i=1
the entire algorithm at once
m = A[1]
is the most common reason
loop
for panicking and drowning.
“loop-invariant:
exit when (i=n)
m = max(m,A[i+1])
i=i+1
endloop
return(m)
“postCond:
<assertion0: x is an integer> How do we prove that
if( <x is odd> ) then this block of code works?
x=x+1
What is the definition of “works”?
else
nothing Assertion:
end if Any statement of what must
<assertion1: x is even> be true when at this spot.
If the computation meets the assertion
when at the beginning of a block,
then it should meet the assertion
when at the end of a block.
If the computation does not meet the assertion

when at the beginning of a block,
then it should display an error message.
14
<assertion0: x is an integer> How do we know the LHS is true?
if( <x is odd> ) then
x=x+1 There are two cases:
else If the LHS is false,
nothing then we don’t care..
end if
<assertion1: x is even>
Defn of Correctness: <assertion0>

<assertion1>
code
Deduction Format:
Deduction Goal: LHSRHS
______ LHS Assumption
______ …
______ RHS Proved somehow
LHSRHS Deduction Conclusion 15
x=x+1 There are two cases:
else If the LHS is true,
nothing then we need to prove the RHS.
end if We can assume LHS is true, because
<assertion1: x is even> it is the case we are considering.

<assertion1>
code
Deduction Fomat:
______ …
x=x+1 In Math1090, Jeff has an Oracle
else help you assume the LHS.
nothing
end if

<assertion1>
code
Deduction Fomat:
______ …
x=x+1 In EECS3101, Jeff has Fairy Godmother
else grant that the LHS is true.
nothing
end if

<assertion1>
code
Deduction Fomat:
______ …
x=x+1 Or equivalently Jeff says that
else we fly in from Mars and are
nothing assured only that the LHS is true.
end if

<assertion1>
code
Deduction Fomat:
______ …
α
Actions
β
vs Assertions
<assertion0: x is an integer> But there are two paths
if( <x is odd> ) then through the code!
x=x+1
else Cases Goal: γ.
nothing 1. Cases α or β Proved somehow
end if 2. Case 1:
<assertion1: x is even> 3. α Assumption
Proved somehow
4. ….
5. γ Assumption
6. Case 2:
7. β Proved somehow
Cases Conclusion
8. ….
20
α
Actions
β
vs Assertions
x=x+1
else Cases Goal: γ.
nothing 1. Cases α or β Proved somehow
end if 2. Case 1:
<assertion1: x is even> 3. α Assumption
WeWeOur
checkgoal
provetheisγcode
to
= Proved somehow
prove
and<assertion
see γ 1=>are
there
Consider 4. ….
<assertion
We
only
is prove
two
themet β.γ1>=
paths
when
path α. 5. γ Assumption
<assertion
denoted
following 1> β.
α path
and α. 6. Case 2:
is met
We whenγ =
proved 7. β Proved somehow
following
<assertionpath β. Cases Conclusion
1>
in all the cases.
8. ….
21
α
Actions
β
vs Assertions
x=x+1
else Let x0 denote the value of x at assertion0.
nothing Let x1 denote the value of x at assertion1.
end if Case path α:
<assertion1: x is even> x is an integer
0
x0 is odd x1 is even
x1=x0+1
Jeff says that we fly
in from Mars and are
assured only that
<assertion0>
is true.
22
α
Actions
β
vs Assertions
x=x+1
0
x1=x0+1
In case path α,
the if condition is true
and the first code is executed
giving this relationship
between x0 and x1.
23
α
Actions
β
vs Assertions
x=x+1
0
x1=x0+1
Our goal is to prove that
when the computation gets to it,
<assertion1> is true.
24
α
Actions
β
vs Assertions
x=x+1
0
x1=x0+1
We then argue that if we assume
the LHS, we can prove the RHS.
25
α
Actions
β
vs Assertions
x=x+1
end if Case path β:
0
¬<x0 is odd> x1 is even
x1=x0
Jeff says that we fly
in from Mars and are
assured only that
<assertion0>
is true.
26
α
Actions
β
vs Assertions
x=x+1
0
x1=x0
In case path β,
the if condition is not true
and the second code is executed
giving this relationship
between x0 and x1.
27
α
Actions
β
vs Assertions
x=x+1
0
x1=x0
Our goal is to prove that
when the computation gets to it,
<assertion1> is true.
28
α
Actions
β
vs Assertions
x=x+1
0
x1=x0
We then argue that if we assume
the LHS, we can prove the RHS.
29
α Actions vs Assertions
<assertion0>
if( <condition1> ) then Definition of Correctness
code<1,true>
else <assertion0>
code<1,false> any <conditions> <assertionr>
end if code
<assertion1>
…
Each path through this code

must be check.
<assertionr-1>
How many such paths are there?
if( <conditionr> ) then There are 2r paths!
code<r,true>
else Is there a faster way?
code<r,false> Break up the code Each sub-path goes from
end if with more assertions. one assertion to the 30
α β
<assertion0>
if( <condition1> ) then Case path α: Step 1
code<1,true> <assertion0>
else <condition1> <assertion1>
code<1,false> code<1,true>
<assertion1> <assertion0>
<assertion1>
…
¬<condition1>
<assertionr-1> code<1,false>
if( <conditionr> ) then By cases:
code<r,true> <assertion0>
else <assertion1>
code
code<r,false>
Each sub-path goes from
end if
one assertion to the 31
<assertion0>
if( <condition1> ) then Case path α: Step r
code<1,true> <assertionr-1>
else <conditionr> <assertionr>
code<1,false> code<1,true>
<assertion1> <assertionr-1>
<assertionr>
…
¬<conditionr>
α β
<assertionr-1> code<1,false>
if( <conditionr> ) then By cases:
code<r,true> <assertionr-1>
else
<assertionr>
code
code<r,false>
end if
<assertion0>
if( <condition1> ) then
Sequentiual
<assertion0>
else code
code<1,false> <assertion1>
<assertion2>
end if code
<assertion1> Note <assertion1> is both the end of one
…
black and the beginning of the next.

<assertionr-1>
if( <conditionr> ) then
code<r,true>
else By ??
transitivity:
code<r,false> <assertion0>
end if
<assertion
Each sub-path 2>
goes from
code one assertion to the 33
<assertion0>
Sequentiual
<assertion0>
else code
code<1,false> <assertion1>
<assertion2>
end if code
<assertion1>
…
…
<assertionr-1>
if( <conditionr> ) then <assertionr-1>
<assertionr>
code<r,true> code
else By transitivity:
code<r,false> <assertion0>
end if
<assertionr>
code
34
<assertion0>
code<1,true>
else
code<1,false> Must check 2r not 2r different
sub-paths through the code.
end if
<assertion1>
<assertionr-1>
if( <conditionr> ) then
code<r,true>
else
code<r,false>
end if
Max( a,b,c ) Actions vs Assertions
“preCond:
Input has 3 numbers.”
m=a
“assert: m is max in {a}”
if( b>m )
m=b
endif
<preCond>
“assert: m is max in {a,b}” <assertion0>
if( c>m ) code
m=c
endif
“assert: m is max in {a,b,c}”
return(m)
“postCond:
return max in {a,b,c}”
36
“preCond:
m=a
if( b>m ) <assertion0>
m=b <assertion1>
endif <condition1>
“assert: m is max in {a,b}” code<1,true>
if( c>m )
m=c <assertion0>
endif ¬<condition1> <assertion1>
“assert: m is max in {a,b,c}” code<1,false>
return(m)
“postCond:
37
“preCond:
m=a
if( b>m ) <assertion1>
m=b <assertion2>
endif <condition2>
“assert: m is max in {a,b}” code<2,true>
if( c>m )
m=c <assertion1>
endif ¬<condition2> <assertion2>
“assert: m is max in {a,b,c}” code<2,false>
return(m)
“postCond:
38
“preCond:
m=a
if( b>m )
m=b
endif
<assertion2>
“assert: m is max in {a,b}” <postCond>
if( c>m ) code
m=c
endif
“assert: m is max in {a,b,c}”
return(m)
“postCond:
39
Max( A )
“preCond:
of n values.” Definition of Correctness
i=1 <preCond>
m = A[1] any <conditions> <postCond>
loop code
“loop-invariant: Each path through this code
m is max in {A[1]..A[i]}” must be check.
exit when (i=n) How many such paths are there?
There are ∞ paths!
m = max(m,A[i+1])
i=i+1 Is there a faster way?
endloop Break up the code Each sub-path goes from
return(m) with more assertions. one assertion to the
“postCond: next.
Only 3 sub-paths!
Assertions:
– Useful for thinking
developing
describing
proving correctness
about algorithms
41
Assertions:
– An assertion is a statement/picture
about the current state of the data structure
that is either true or false.
eg. the amount in your bank account is not negative.
– An assertion is not a task for the algorithm to perform.
– It is only a comment that is added
for the benefit of the reader
and for the writer!
– To make sure no one gets lost.
42
Assertions:
– As long as the input meets the precondition,
the assertion should be true independent of
• the path followed through the code
• and the input.
– If it is false,
then something has gone wrong in the logic of the algorithm.
43
Assertions:
– is a function A from current state to true/false
Eg A = “x is odd”
A(<x=5, y=3, line84> ) = True  Computation is
on the path
A(<x=8, y=3, line84> ) = False  Something has
Current state of the computation: gone wrong!
– Everything that is true at a particular instant in time.
– Imagine flying in from Mars,
what do you need to know about the present
to be able to continue.
– It does not say anything about the past,
i.e. how the computation got here.
– Eg <x=8, y=3, current line of code = line 84> 44
Assertions:
Don't Be Frightened
An assertion does not need to
consist of formal mathematical mumble jumble.
Use an informal description and a picture.
45
Algorithm Termination
• Computation steadily approaches goal
79 km 75 km
You need to define some

Measure of progress
to prove that your algorithm
eventually terminates. 46
Algorithm Termination
You need to define some
Measure of progress
to prove that your algorithm
eventually terminates.
• An Measure of Progress, M
– is a function
• that takes as input the current state
• and that outputs a real number
– Eg Measure = “value of y”
• M(<x=5, y=3, line84> ) = 3
– We must prove that this number goes down (up)
each iteration. 47
& Loop Invariants loop (until done)
take step
end loop
• A good way to structure many

programs:
• Store the key information you
currently know in some data
structure.
• In the main loop, take a step
forward towards destination
• by making a simple change to
this data.
48
<preCond>
codeA Iterative Algorithms
loop
<loop-invariant>
exit when <exit Cond>
codeB
codeC
<postCond>

1
0 T+
i-1 i
Trust who passes you
the baton
next i
and go around once
i
49
<preCond>
loop
<loop-invariant>
codeB
codeC
<postCond>

50
<preCond>
loop
<loop-invariant>
Loop Invariants
codeB
codeC
<postCond>

A loop invariant is a
5 km statement/picture
about the state of your computation
9 km to make sure it does not get lost.
Your algorithm must only maintain it
while making progress 51
Loop Invariants
Paradigm Shift
52
An Algorithm viewed
as a sequence of
Loop Invariants
Actions: Landmarks: ie an Assertion
of the static
picture
of what is true
.. Straight at that moment
.. Right in the journey.
.. Straight
.. Left …..
I can get lost!!! 53

An Algorithm viewed
as a sequence of
Loop Invariants
Actions: Landmarks:
m=4
m = max(m,5)
m = max(m,3)
m = max(m,7)
m = max(m,1)
I can get lost!!! 54

An Algorithm viewed
as a sequence of Loop Invariants
Assumptions
Actions: Landmarks: (picture)
One step at a time: that must be true.
Input:
Move from previous landmark
4 5 3 7 1
m=4
Max of 4 is 4
m = max(m,5)
Max of 4 5 is 5
m = max(m,3)
Max of 4 5 3 is 5
m = max(m ,7 )
Max of 4 5 3 7 is 7
m = max(m,1)
Max of 4 5 3 7 1 is 7
With landmarks, Output: Max is 7.
I don’t got so lost. 55
Max( A ) Preconditions:
“preCond: Any assumptions that must be
Input is array A[1..n] true about the input instance.
of n values.”
Input:
4 5 3 7 1
Max of 4 5 3 is 5
Output: Max is 7.
56
Max( A ) Postconditions:
“preCond: The statement of what must be
Input is array A[1..n] true when the
of n values.” algorithm/program returns.
Input:
4 5 3 7 1
Max of 4 5 3 is 5
“postCond:
Output: Max is 7.
return max in{A[1]..A[n]}” 57
Max( A ) Loop Invariant:
“preCond: any assumptions (picture) that
Input is array A[1..n] must be true at the top of the loop.
of n values.”
Input:
“loop-invariant: 4 5 3 7 1
Max of 4 5 3 is 5
“postCond:
Output: Max is 7.
Max( A ) Establish
“preCond: the loop invariant.
of n values.”
i=1
m = A[1]
Input:
m is max in {A[1]..A[i]}” Max of 4 is 4
Max of 4 5 3 is 5
“postCond:
Output: Max is 7.
Max( A ) Establish
of n values.”
i=0
m = 0 -
Input:
“loop-invariant: -44 5 3 7 1
m is max in {A[1]..A[i]}” Max of  is ?

m is the smallest number that is
bigger an every element in the Max of 4 5 3 is 5
empty set.
“postCond:
Output: Max is 7.
Max( A ) Maintain
of n values.”
i=1
m = A[1]
loop Input:
m = max(m,A[i+1])
i=i+1 Max of 4 5 3 is 5
endloop Max of 4 5 3 7 is 7
“postCond:
Output: Max is 7.
Max( A ) Obtaining
“preCond: Postcondition
of n values.”
i=1
m = A[1]
loop Input:

exit when (i=n)
m = max(m,A[i+1])
i=i+1 Max of 4 5 3 is 5
endloop Max of 4 5 3 7 is 7
return(m) Max of 4 5 3 7 1 is 7
“postCond:
Output: Max is 7.
Max( A )
“preCond: Iterative Algorithms
Input is array A[1..n] Loop Invariants
of n values.”
i=1 A loop invariant is
m = A[1] any assumptions
loop (picture)
“loop-invariant: that must be true
m is max in {A[1]..A[i]}” at the top of the loop.
exit when (i=n)
How can you possibly understand
m = max(m,A[i+1]) this algorithm without knowing
i=i+1 what is true when
endloop the computation is here?
return(m)
“postCond: Tell me!
Max( A )
of n values.”
i=1 A loop invariant is
m = A[1] • Not just to please the prof.
loop • Not just to prove correctness.
“loop-invariant: • Use them to think about your
m is max in {A[1]..A[i]}” algorithm!
exit when (i=n) • Before you start coding.
m = max(m,A[i+1]) • What do you want to be
i=i+1 true in the middle of your
endloop computation?
return(m) – You need to know.
“postCond: – Let your reader know.
Max( A )
of n values.”
i=1 Definition of Correctness
m = A[1] <preCond>
loop <any conditions> <postCond>
“loop-invariant: code
exit when (i=n) array A[1..n] m is max
m = max(m,A[i+1]) in {A[1]..A[n]}
i=i+1 code
endloop
return(m) How is this proved?
“postCond:
Max( A )
of n values.”
i=1 Step 0
m = A[1]
<preCond> <loop-inv>
loop
codeA
“loop-invariant:
exit when (i=n)
m = max(m,A[i+1]) array A[1..n]
m is max
i=i+1 in {A[1]..A[i]}
endloop i=1
return(m) m = A[1]
“postCond:
Max( A )
of n values.”
i=1
m = A[1]
loop
“loop-invariant:
m is max in {A[1]..A[i]}” Step 1
exit when (i=n) <loop-invariantt>
m = max(m,A[i+1]) ¬<exit Cond> <loop-invt+1>
i=i+1 codeB
endloop m is max m is max
return(m) in {A[1]} in {A[1],A[2]}
“postCond: m = max(m,A[2])
Max( A )
of n values.”
i=1
m = A[1]
loop
“loop-invariant:
m is max in {A[1]..A[i]}” Step 2
i=i+1 codeB
return(m) in {A[1],A[2]} in {A[1]..A[3]}
“postCond: m = max(m,A[3])
Max( A )
of n values.”
i=1
m = A[1]
loop
“loop-invariant:
m is max in {A[1]..A[i]}” Step i
i=i+1 codeB
return(m) in {A[1]..A[i]} in {A[1]..A[i+1]}
“postCond: m = max(m,A[i+1])
Max( A )
of n values.”
i=1 Induction
m = A[1]
loop
“loop-invariant:
i=i+1 codeB
return(m) in {A[1]..A[it]} in {A[1]..A[it+1]}
“postCond: mt+1 = max(mt,A[it+1])
Max( A )
of n values.”
i=1
m = A[1] Last Step
loop
“loop-invariant: <loop-invariant>
<exit Cond>
m is max in {A[1]..A[i]}” <postCond>
exit when (i=n) codeC
m = max(m,A[i+1])
i=i+1 m is max
endloop in {A[1]..A[i]} return max
return(m) i=n in {A[1]..A[n]}
“postCond: return(m)
Max( A )
of n values.”
i=1 Definition of Correctness
m = A[1] <preCond>
loop <any conditions> <postCond>
“loop-invariant: code
exit when (i=n) array A[1..n] return is max
m = max(m,A[i+1]) in {A[1]..A[n]}
i=i+1 code
endloop
return(m)
“postCond:
Max( A )
of n values.”
i=1 Establishing the loop invariant:
m = A[1] preCond
loop-invariant
loop code
“loop-invariant: Exit
Maintaining the loop invariant:
exit when (i=n) loop-invariantt-1
m = max(m,A[i+1]) ¬exit Cond loop-invariantt
codeB
i=i+1
endloop Exit Obtaining Postcondition:
return(m) loop-invariant 
“postCond: exit Cond postCond
return max in {A[1]..A[n]}” code 73
Loop Invariants
Establishing the loop invariant:
preCond
From these we code
need to prove
Exit
that the code Maintaining the loop invariant:
maintains the loop-invariantt-1
loop invariant.
¬exit Cond loop-invariantt
codeB
Exit Obtaining Postcondition:

loop-invariant 
exit Cond postCond
code 74
Let S(i) be a yes/no statement, for each positive integer i.
• S(i) ≡ “It will rain on the ith day”
From these we
need to prove
that the code
maintains the
loop invariant.
S(0) S(1) S(2) S(3) S(4) S(2) S(3) S(4) …
75
Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)
• S(i) ≡ “My iterative algorithm maintains its loop invariants
for the first i iterations.”
preCond
From these we codeA
loop
need to prove loop-invariant
that the code exit when exit Cond
codeB
maintains the endloop
loop invariant. codeC
postCond
My goal is to prove "i

S(i).
There are an infinite number of different i.
How can you prove S(i) for all of them???
76
Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)
S(i).
My first step is to prove S(0).

preCond
code
Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)
S(i).
Then I prove "i [S(i-1)S(i)].
Exit
loop-invariantt-1
codeB
Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)
S(i).
Then I prove "i [S(i-1)S(i)].
To prove this using the game,

I first give you an arbitrary i.
I help you by assuring you of S(i-1).
I must prove S(i).

Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)
S(i).
Then I use modus ponens to get "i S(i).
Why can I then conclude "i S(i)???

Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)

S(i).
S(0) S(1) S(2) S(3) S(4)

…
We will colour S(i) by S(i) when we know it is true.
S(i) when false.
Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)
Assume the following:
S(0) & "i S(i-1)S(i)]
I can help you with these.
S(i).
Telling me that S(0) is true is a good start.
S(0) S(1) S(2) S(3) S(4)

…
Induction
Induction Axiom:
[S(0) & "i [S(i-1)S(i)]]  "i S(i)
Assume the following:
S(0) & "i S(i-1)S(i)]
I can help you with these.
S(i). 342
How do I prove S(1)?
342 231 342
I will set i=1 and assure you that S(0)S(1).
231 342
Excellent. We know S(0) and this gives us S(1).
S(0) S(1) S(2) S(3) S(4)
…
Each positive integer is eventually reached!
"i $ time t (i is reached in time t)
Logic Game
Max( A )
“preCond:
Input is array A[1..n] preCond
of n values.” code
i=1
m = A[1] Exit
loop Maintaining the loop invariant:
“loop-invariant: loop-invariantt-1
m is max in {A[1]..A[i]}” ¬exit Cond loop-invariantt
exit when (i=n) codeB
m = max(m,A[i+1])
i=i+1 Exit Obtaining Postcondition:
endloop loop-invariant 
return(m) exit Cond postCond
“postCond: code
Logic Game
preCond: Input is array A[1..n] of n values.”
loop-invariant: m is max in {A[1]..A[i]}”
" Situations preCond
code
Input: 4 5 3 7 1
Being an assumption, I can assure you that the

precondition is true for the input,
ie Input is array A[1..n] of n values.
But he does not know anything else.

The implied " means I choose.
85
Logic Game
preCond
code
Input: 4 5 3 7 1
loop-invariant0 Max of 4 is 4
i=1 & m=4
My goal is to prove that the code before the loop

• Establishes the loop invariant.
Though I get to write this code,
I will instead, first choose what the data structure will look like.
86
Logic Game
preCond
code
Input: 4 5 3 7 1
loop-invariant0 Max of 4 is 4
i = last index considered
index = 1
I now must write code to go from the one state to the other.
i=1
m = A[1]
87
Logic Game
Exit
codeB
Input: 4 5 3 7 1
Being an assumption, I can assure you that we are at

some arbitrary point in the computation and
the loop invariant is true.
m is max in {A[1]..A[i]}
This tells me that we have two variables m and i and

m is the maximum of the first i input values.
88
Logic Game
Exit
" Situations loop-invariantt-1
codeB
Input: 4 5 3 7 1
loop-invariantt-1 Max of 4 5 3 is 5
But he does not know anything else.

Eg He does not know # of iterations t-1.
The implied " means I choose.
89
Logic Game
Exit
codeB
Input: 4 5 3 7 1
loop-invariantt Max of 4 5 3 7 is 7
My goal is to prove that after the code goes around the loop
• Progress has been made
• The loop invariant is true again.
Though I get to write the code for this loop,
I will instead, first choose what the data structure will look like.
90
Logic Game
Exit
codeB
Input: 4 5 3 7 1
loop-invariantt Max of 4 5 3 7 is 7
I now must write code to go from the one state to the other.
• m is max in {A[1]..A[i]}
• The next input value to look at is A[i+1].
• The new value of m should be the max of these,
ie m = max(m,A[i+1])
• Then we change i, ie i = i + 1
91
Logic Game
postCond: return max in {A[1]..A[n]}”
Obtaining Postcondition:
Exit loop-invariant 
code
Input: 4 5 3 7 1
loop-invariant Max of 4 5 3 is 5
My goal is to prove that in the end, the post condition is true,
ie return max in {A[1]..A[n]}
I can assure you that the loop invariant is true,

ie m is max in {A[1]..A[i]}
If i=n, then the post condition and the loop invariant are the same!
Let’s make i=n the exit condition. 92
Logic Game
postCond: return max in {A[1]..A[n]}”
Obtaining Postcondition:
Exit loop-invariant 
code
Input: 4 5 3 7 1
loop-invariant Max of 4 5 3 is 5
I can assure you that the loop invariant is true,

ie m is max in {A[1]..A[i]}
exit Cond: i=n. code: Return m.
My goal is to prove that in the end, the post condition is true,

ie return max in {A[1]..A[n]}
93
Code vs
Math Assertions
I don’t like i=i+1
It is code/actions!
If you want to prove things,
you need math assertions.
it+1 = it+1
I will try to translate.
Let's demonstrate
with a strange
example.
94
Your are at the top of the loop.
You do not know what
Code vs
happened before. Math Assertions
Let xt and yt be the values of x and y
at the beginning of the iteration.
xt yt
loop
x y
4: “loop-invariant: ??
5: exit when ??? Does this mean that Could be denoted
6: x = x + 1 this is the t th
iteration?
x´ and y´
7: y = y + x
No! In order be able to
8: x = x  y do it only once, Don’t consider
end loop you can’t know what
happened before. t-1 or t+1
95
Code vs
Let xt+1 and yt+1 be their values after going around again.
( xt +1)(yt+xt+1) yt +xt+1
loop
x y
xt+1 = (xt+1)(yt+xt+1)
5: exit when ???
yt+1 = yt+xt+1
6: x = x + 1
7: y = y + x Excellent.
8: x = x  y These are math assertions giving
end loop the relationship between
the old and new values.
96
Code vs
Let xt+1 and yt+1 be their values after going around again.
( xt +1)(yt+xt+1) yt +xt+1
loop
x y
xt+1 = (xt+1)(yt+xt+1)
5: exit when ???
yt+1 = yt+xt+1
6: x = x + 1
7: y = y + x Let's build a DFA from 2001
(Deterministic Finite Automata)
8: x = x  y
end loop δ( qline=4, x=xt, y=yt , iteration)
= q line=4, x=(x +1)(y +x +1), y=y +x +1
t t t t t
97
loop
“loop-invariant: ??
Code vs
exit when ???
x=x+1
Math Assertions
y=y+x
x=xy <loop-invariantt>
end loop
¬<exit Cond> <loop-invt+1>
codeB
<loop-invariantt>
Induction
true about xt and yt
¬<exit Cond> <loop-invariantt+1>
true about xt and yt
true about xt+1 and yt+1
codeB Assume LI is true after t iterations
xt+1 = (xt+1)(yt+xt+1)
and prove true after t+1
yt+1 = yt+xt+1
98
A Relay Race
0
T+
1
• Your job is only to
i-1 i • Trust who passes you the baton
• Go around once
i • Pass on the baton
i
99
Person 0: Carries baton region to safe region
Establishing Loop Invariant
<preCond>
<loop-invariant>
codeA
<preCond>
0
codeA
loop
<loop-invariant>
codeB
endloop
codeC
<postCond>
100
Person i: Running around the track
Exit Maintaining Loop Invariant

<loop-invarianti-1>
¬<exit Cond> <loop-invarianti>
codeB <preCond>
i-1
codeA
i
loop
<loop-invariant>
codeB
endloop
codeC
<postCond>
i
101
Last Person: Carries baton from safe region to finish.
Clean up loose ends
<loop-invariant>
Exit
<exit Cond> <postCond>
codeC <preCond>
T+
1
codeA
loop
<loop-invariant>
codeB
endloop
codeC
<postCond>
102
Partial Correctness
<preCond>
<loop-invariant>
codeA
Maintaining Loop Invariant
Exit
<loop-invariantt>
codeB
Clean up loose ends
<loop-invariant>
Exit
codeC
Proves that IF the program terminates then it works
<PreCond> & <code> Þ <PostCond> 103
Designing an Algorithm
Define Problem Define Loop
Invariants
Define Step Define Exit Condition Maintain Loop Inv

Exit
Exit
Initial Conditions Ending
Exit
104
Physics( )
“preCond: Conservation of Energy
Input is a roller coaster
with car starting high.”
loop
exit when (car at end) Actions?

→ →
x t+t,v t+t Too much computation!
→ →
= Newton(x t,v t) How does Newton solve this?
endloop What is conserved?
Energy & Mass
return(vfinal) Et+t = Et
“postCond: (kinetic  potential)
105
Physics( )
loop
“loop-invariant:
Et = E0”
A loop invariant is
exit when (car at end)
→ → any assumptions
x t+t,v t+t
→ → (picture)
= Newton(x t,v t)
that must be true
endloop
at the top of the loop.
return(vfinal) Et+t = Et
“postCond: 106
Physics( )
loop
“loop-invariant:
Et = E0”
A loop invariant is
→ → like property
→ → in physics that is
conserved/maintained.
endloop
return(vfinal)
“postCond: 107
Physics( )
loop
“loop-invariant:
Et = E0”
→ →
x t+t,v t+t <loop-invariantt>
→ →
¬<exit Cond> <loop-invt+1>
codeB
endloop
Et+t = Et = E0
return(vfinal)
“postCond: 108
Physics( )
E0 = ½mv02 + h0m
loop
“loop-invariant:
Et = E0”
<preCond> <loop-inv>
codeA
endloop
return(vfinal)
“postCond: 109
Physics( )
E0 = ½mv02 + h0m
loop
“loop-invariant:
Et = E0”
<loop-invariant>
x t+t,v t+t <postCond>
<exit Cond>
= Newton(x t,v t) codeC
endloop
vfinal = (2E0/m-2hfinal)½
return(vfinal)
“postCond: 110
Physics( )
E0 = ½mv02 + h0m
loop
“loop-invariant:
Et = E0”
endloop
vfinal = (2E0/m-2hfinal)½
return(vfinal)
“postCond: 111
Invariants

Exit
Exit
Exit
112
Differential Equations
Consider a block bouncing on a spring.
We will solve the differential equation expressing its movement.
T T
Let x(t) denote its distance from rest.

It will bounce from x(t)=+xm to -xm.
113
T T
It is at top peak,
has stopped moving,
and is accelerating down.
114
T T
It is in the center,
is moving fast down,
and is not accelerating.
115
T T
It is at bottom peak,
has stopped moving,
and is accelerating up.
116
T T
It is in the center,
is moving fast up,
and is not accelerating.
117
T T
It is at top peak,
it has stopped moving,
and is accelerating down.
118
Acceleration(x) = -x(t)
T T
Let x(t) denote its distance from rest. Newton: F=MA

It will bounce from x(t)=+xm to -xm.
Acceleration(x) of block on spring Pull the string farther,
= Force / Mass it pulls back harder
= (-k x(t))/ Mass in the opposite
direction. 119
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
Let S(T) be a yes/no statement, for each positive “real” T.
loop S(T) ≡ “We have plotted distance x(t) from 0 to T
inv: and know x(T) and velocity(T).”
T T
Dude! You can’t do induction on the reals!

They are not countable.
Let's do induction just on multiples of some infinitesimal
ε. These are countable.
T = 0ε, 1ε, 2ε, 3ε, …
120
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T
My goal is to plot x(t) for t≥0.

It is sufficient to prove "T S(T),
ie for every T, we have plotted up to T.
If we use induction, it is sufficient to prove
S(0) & "i [S(i-1)S(i)] 121
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T
S(0) simply states that we know

the initial location and velocity of the block.
x(0)=+xm
Pull the block back
velocity(0)=0 and let go.
Establishing the loop invariant.
122
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T
My goal now is to prove

"T [S(T)S(T+ε)]
To prove this using the game,
I first give you an arbitrary T.
123
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T
T +ε
I help you by assuming S(T).

• x(t) has been plotted up to T. Exit
Then I prove S(T+ε). Maintainin

•I need only need to plot from T to T+ε. g
• loop inv124
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T, T +ε
T +ε
I help you by assuming S(T).

• x(t) has been plotted up to T. Exit
• We know x(T) and velocity(T).
Then I prove S(T+ε). Maintainin
•I need only need to plot from T to T+ε. g
• loop inv125
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T, T +ε
T +ε
•x(T+ε) = x(T) + velocity(T)  ε

(cm) = (cm) + (cm/sec)  (sec)
It is so important to balance units.
126
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T, T +ε
T +ε

If I travel this fast for this amount of time

then this is how far I travel.
127
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T, T +ε
T +ε

If I am here and travel this far

then this is where I will end up. 128
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T, T +ε
T +ε
•x(T+ε) = x(T) + velocity(T)  ε Hey! You are

(cm) = (cm) + (cm/sec)  (sec) assuming that
during [T,T+ε]
that the velocity
Yes, but is an infinitesimal. is a constant.
129
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T, T +ε
T +ε

• acceleration(T) = - x(T)
• velocity(T+ε) = velocity(T) + acceleration(T)  ε
(cm/sec) = (cm/sec) + (cm/sec2)  (sec)
130
Induction Axiom:
[S(0) & "T [S(T)S(T+ε)]]  "T S(T)
T T, T +ε
T +ε
This completes the proof.
Thanks for your help.

But we just solved it on multiples of ε.
True.
But we can now take the limit as ε goes to zero. 131
Invariants

Exit
Exit
Exit
132
Life( me)
“preCond: Game of Life
I am born.”
Hopefully my parents help
loop
“loop-invariant:
I am well
and reasonably on track” One day at a time.
exit when (I die)
Maintain the loop invariant
Make a little progress
Make the world a little better.
endloop
next
“postCond:
It was a well spent and good life”
133
Invariants

Exit
Exit
Exit
134
Define Problem Define Loop Define Measure of
Invariants Progress
79 km

Exit
Exit
Make Progress Initial Conditions Ending

Exit
79 km 75 km
km 0 km Exit Exit
135
I like understanding things
using a story.
I will now tell one.
136
The Getting to School Problem
137
Problem Specification
• Pre condition: location of home and school
• Post condition: traveled from home to school
138
Algorithm
• What is an Algorithm?
139
Algorithm
• The algorithm defines the computation route.
140
Algorithm
• Is this reasonable?
141
Complexity
• There are an infinite number of input instances
• Algorithm must work for each.
142
Complexity
• Difficult to predict where computation might be
in the middle of the computation.
143
Location of Computation
• The current “State” of computation
is determined by values of all
variables.
144
• Suppose the computation ends up here?
145
• Suppose the computation ends up here?
146
Don’t Panic
• Where ever the computation might be,
take best step towards school.
good enough
147
General Principle
• Do not worry about the entire computation.
• Take one step at a time!
next
148
Defining Algorithm
• Wherever the computation might be,
take step towards school.
149
Take a step
• What is required of this step?
150
A Measure of Progress
75 km
79 km
151
Defining Algorithm
• Is this sufficient to define a working algorithm?
79 km 75 km
152
Working Algorithm
• Computation steadily approaches goal
79 km 75 km
153
Defining Algorithm
• Extra requirements
79 km 75 km
79 km
km 0 km
78.999 km
154
Define a Step
• Wherever the computation might be,
take best step towards school.
155
Computation Stuck
• Some location too confusing for algorithm
• Algorithm too lazy to define step for every
location.
156
Safe Locations
• Algorithm specifies from which locations
it knows how to step.
157
Loop Invariant
• “The computation is presently in a safe
location.”
• Maybe true and maybe not.
If not something has gone wrong.
158
Defining Algorithm
• From every safe location,
define one step towards school.
159
Take a step
• What is required of this step?
160
Maintain Loop Invariant
• If the computation is in a safe location,
it does not step into an unsafe one.
161
• Can we be assured that the computation

will always be in a safe location?
162
• Can we be assured that the computation

will always be in a safe location?
No. What if it is not initially true?

163
From the Pre-Conditions on the input instance

we must establish the loop invariant.
164
• Can we be assured that the

computation will always be
in a safe location?
• By what principle?
165
• By Induction the computation will
always be in a safe location.
 S (0) 


   iS ( i ) 

  iS ( i )  S ( i  1 ) 
166
Ending The Algorithm
Define Exit Condition Exit
Termination:
With sufficient progress,
0 km Exit
the exit condition will be met.
When we exit, we know

• exit condition is trueExit
• loop invariant is true Exit
from these we must establish

the post conditions.
167
Let’s Recap
168
Designing an Algorithm Is this sufficient?
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
169
Consider an Algorithm
Exit
Exit
79 km 75 km km 0 km Exit Exit
170
Loop Invariant Maintained
Exit
171
Computation can always proceed
Exit
172
Computation always makes progress
79 km 75 km
Exit
79 km 75 km
173
Computation Terminates
km
0 km
0 km Exit
79 km 75 km
0 km Exit
174
Computation Terminates
Exit
Exit
175
Consider an Algorithm
Exit
Exit
79 km 75 km km 0 km Exit Exit
This is sufficient!
176
Code from LI
It is hard to design an algorithm
(or even worse write its code),
before you understand
the proof of why the algorithm works.
It is hard to proof of the algorithm works,

before you know the code.
We will
• First design the loop invariant
• From it, we will follow mechanical steps,
ie minimal thinking
and the code will fall out.
We will do an easy example.
but the same ideas can help with many problems.
177
Code from LI
• Precond: The input is an integer I.
• Postcond: Output = S = 12 + 22 + 32 + … +
I2
178
Code from LI
I2 What is the loop invariant?
Each iteration add in a new object.
Action not picture
If you fight me,
I will fail you
179
Code from LI
I2 What is the loop invariant?
What do you want to be true in the middle of the computation?
The post condition is that a set of In the middle of the computation
things have been added together. some prefix of them
have been added together!
i could denote the index of
next object to add in.
Hence, the number added
so far would be i-1.
180
Code from LI
I2 + 2i2
+ (i-1)
• LoopWhat
Invariant:
do you want tos be
= true
12 +in2the
2
+middle
32 + of…the computation?
The post condition is that a set of In the middle of the computation
things have been added together. some prefix of them
have been added together!
i could denote the index of or i could denote
next object to add in. the number added.
Hence, the number added Hence, the next to add in
so far would be i-1. would be the i+1st .
It is a matter of esthetics. Though it changes everything.
All the matters is that
you are consistent! 181
Code from LI
I2 + i2
• Loop Invariant: s =I =1input
2
+ 22 + 32 + …
S = output
i,s = our current data structure.
The loop invariant tells us a
relationship between i and s.
182
Code from LI
I2 + i2
• Loop Invariant:You might
s =be1tempted
2
+ 22 + 32 + …
to think
about the range of i.
Could it be 0 or start at 1?
Does it end at I-1, I, or I+1?

middle
183
Code from LI
I2 + i2
s =be1tempted
2
+ 22 + 32 + …
to think
184
Code from LI
I2 + i2
s =be1tempted
2
+ 22 + 32 + …
to think
Start in the middle of the computation.

Say i=5.
The fairy god mother assures you the loop invariant is true.
Your job is only to
make progress
and maintain the loop invariant.
185
Code from LI
I2 + i2
• Loop Invariant: s = 1 2 + 22 + 32 + …
• Measure of Progress
79 km
• i = # of objects added.
Exit
79 km 75 km
• Step to make progress
• i = i+1
• Exit Condition
Exit • Don’t worry about it yet. 186
Code from LI
In order to make progress,
we may head in one direction.
Exit
79 km 75 km
• i = i+1
187
Code from LI
Exit
79 km 75 km
• i = i+1
188
Code from LI
Exit
In order to maintain the loop invariant,

we must adjust to come back to the path.
189
Code from LI
This may be how we determine

the step in then next iteration.
190
Code from LI
I2 + i2
• Loop Invariant: s = 1 2 + 22 + 32 + …
• Partial Step: i = i+1
Exit
<loop-invariantt>
codeB
Let it and st be the values of i and s at the

beginning of the iteration and let it+1 and st+1 be
their values after going around again.
191
Code from LI
I2 + i2
• Loop Invariant: s = 1 2 + 22 + 32 + …
• Partial Step: i = i+1 Use the fact that you
<loop-invariantt> already have the LI true
¬<exit Cond> and take a small step.
codeB: Make the loop Invariant true!!!
Solve the entire algorithm!!!
Panic!!
next
<loop-invariantt+1>192
Code from LI
I2 + i2
• Loop Invariant: s =(codeB
12 +in2loop)
2
+ 32 + …
<loop-invariantt> s = 12 + 2 2 + 32 + … + i 2
t t
¬<exit Cond> it = 5
codeB: it+1 = it + 1 (math)
st+1
Write down LHS/RHS
of what we must prove
=12 + 22 + 32 + … + it+12 <loop-invariantt+1>193
Code from LI
I2 + i2
• Loop Invariant: s = 1 2 + 22 + 32 + …
<loop-invariantt> s = 12 + 2 2 + 32 + … + i 2
t t
¬<exit Cond> it = 5
codeB: it+1 = it + 1
st+1 = st + it+12
?? This completes the proof.
st+1 = st + it+12
=[12 + 22 + 32 + … + it2 ]+ it+12
=12 + 22 + 32 + … + it+12 <loop-invariantt+1>194
Code from LI
I2 + i2
• Loop Invariant: s = 1 2 + 22 + 32 + …
s = s + i2 it+1 = it + 1
i=i+1 st+1 = st + it+12
We now translate
i=i+1 this math  code
s = s + i2
195
Code from LI
Sum( I )
“preCond: The input is an integer I”
?
loop
“loop-invariant: s = 12 + 22 + 32 + …+ i2
exit when (i=I)
i=i+1
s = s + i2
endloop
?
“postCond: Output = S = 12 + 22 + 32 + …+ I2”

196
Code from LI
I2 + i2
• Loop Invariant: s ,=s 1=2 s++22i2+ 32 + …
Clean up loose ends
<loop-invariant>
Exit
codeC
197
Code from LI
I2 + i2
<loop-invariant> s = 12 + 2 2 + 32 + … + i 2
<exit Cond> ?? What math condition do I need here ??
codeC: ?? What math relationship do I need here ??
S =12 + 22 + 32 + … + I2 <postCond> 198

Code from LI
I2 + i2
<loop-invariant> s = 12 + 2 2 + 32 + … + i 2
<exit Cond> i=I (math statement)
codeC: s=S (math statement)
I need:
• s=S S is our output
• i=I i is our measure of progress
S =12 + 22 + 32 + … + I2 <postCond> 199

loop while( i≠I ) i<I
“loop-invariant” Code from LI
exit when i=I
code
endloop
What is a math assertion that we what to be true
“i=I”
when we exit? What is the code to make this true?
“postCond”

We now translate
this math  code
I like while statements.
What if I = 7 and I = 4 6 8
It is all scary! 200
loop while( i≠I ) i<I
“loop-invariant” Code from LI
exit when i=I
code
endloop
“i=I”
S=s What is the code to give post condition?
“postCond”

We now translate
this math  code
201
Code from LI
I2 + i2
<preCond>
<loop-invariant>
codeA
i s
i=0
s=0 0 0
1 1
2 1+4=5
3 1+4+9=14 202
Code from LI
Sum( I )
“preCond: The input is an integer I”
i=1
m = A[1]
loop
“loop-invariant: s = 12 + 22 + 32 + …+ i2
exit when (i=I)
i=i+1
s = s + i2
endloop
S=s
return(S)
“postCond: Output = S = 12 + 22 + 32 + …+ I2”
203
Code from LI
I2 + 2i2
+ (i-1)
That was fun. Let’s do it again.
We define i to be
the number added so far. Let’s redefine this to be i-1.
Redo all of the steps to see what changes.
204
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
205
Pointers
206
Pointers
207
Pointers
208
Pointers
209
Pointers
210
Pointers
More on Pointers
211
Pointers
212
Pointers
213
Pointers
Oh dear,
• Getting from A to F is a lot of work.
• This F has one record already changed.
so maybe one iteration has happened already.
Let's roll
backwards one
more iteration.
I like the number

t=0 much more
than t=1.
214
Pointers
215
Pointers
Where should done point?
No! done is of type pointer-to-node.

Not pointer-to-pointer-to-node.
216
Pointers
Where should done point?
None of the nodes are “done”!
Ok
217
Pointers
218
I get nervous. Pointers
Let’s run our generic
loop code once to
make sure it works.
The code needs

to point node 1 at 6.
The code assumes
done is pointing
at node 6.
Ok
219
Pointers
220
Pointers
221
Pointers
222
Pointers
223
Pointers
224
Pointers
225
Pointers
226
Pointers
227
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
228
Different Representations
of Algorithms
• Code
• Running example
• DFA and Turing Machines
• Higher level abstract view
229
Code
Representation of an Algorithm
class InsertionSortAlgorithm extends SortAlgorithm {
void sort(int a[]) throws Exception {
for (int i = 1; i < a.length; i++) {
int j = i;
int B = a[i];
while ((j > 0) && (a[j-1] > B)) {
a[j] = a[j-1];
j--; }
a[j] = B;
}}
Pros and Cons?
230
Code
Pros: Cons:
• Runs on computers • I am not a
• Precise and computer
succinct • I need a higher
• Perception that level of intuition.
being able to code • Prone to bugs
is the only thing • Language
needed to get a job. dependent
(false) 231
Running Example
Try out a problem or
solution on small examples.
232
Running Example
88 52
98
14
31
98 88
25 30
62 23
14
79
233
Running Example
14,23,25,30,31,52,62,79,88,98
Pros and Cons?

234
Running Example
Pros: Cons:
• Concrete • Relies on you to find
• Dynamic the pattern.
• Visual • Does not explain why
it works.
• Demonstrates for
only one of many
inputs.
235
DFA and Turing Machines
Pros and Cons?

236
DFA and Turing Machines
Pros: Cons:
• Good for theorems • Not good for
about algorithms designing algorithms
237
Higher Level Abstract View
1
0 T+
i-1
i
238
Levels of Abstraction
It is hard to think of love in terms of the firing of
neurons.
vs
Software developers view subsystems as entities with

separate personalities, roles, and interactions,
not details of code.
vs
239
Pros: Cons:
• Intuitive for • Mathematical
humans mumbo jumbo
• Useful for • Too abstract
– thinking about • Students resist it
– designing
– describing
algorithms
• View from which 240
Value Simplicity
Abstract away the inessential features of a problem
=
Goal: Understand and think about complex
algorithm in simple ways.
Don’t tune out. There are deep ideas within
the simplicity.
241
Simple Example
Insertion Sort Algorithm
242
Reconsidering Simple Examples
A good martial arts student will attentively repeat each

fundamental technique many times.
In contrast, many college students tune out when a concept

(e.g., depth first search) is taught more than once.
The better students listen carefully in order to refine and

develop their understanding.
Repetition Is An Opportunity Rudich www.discretemath.com 243

Code
class InsertionSortAlgorithm extends SortAlgorithm {
void sort(int a[]) throws Exception {
for (int i = 1; i < a.length; i++) {
int j = i;
int B = a[i];
while ((j > 0) && (a[j-1] > B)) {
a[j] = a[j-1];
j--; }
a[j] = B;
}}
244
5 km
9 km
245
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
246
Problem Specification
• Precondition: The input is a list of n values
with the same value possibly repeated.
• Post condition: The output is a list consisting
of the same n values in non-decreasing order.
88 52
14
31 14,23,25,30,31,52,62,79,88,98
25 98 30
62 23
79
247
• “State” of computation determined
by values of all variables.
248
• “State” of computation determined
by values of all variables.
88 52 14
31 25 62 30 14,23,25,30,31,52,62,79,88,98
98 23
79
88 14<52
31 25<62 30
23
79<98
249
Computation Stuck
• Location too confusing for algorithm
• Algorithm too lazy to define step for
every location.
88 52 14
31 25 62 30 14,23,25,30,31,52,62,79,88,98
98 23
79
88 14<52
31 25<62 30
23
79<98
250
Safe Locations
• Algorithm specifies from which locations
it knows how to step.
88 52 14
31 25 62 30 14,23,25,30,31,52,62,79,88,98
98 23
79
251
Define Loop Invariant
• Loop Invariant:
• Input numbers split to left and right.
• Numbers on left are sorted.
88 52 14
31 25 62 30 14,23,25,30,31,52,62,79,88,98
98 23
79
30 14
23,31,52,88 25 62
98
79
252
Location Not Determined
Which subset of the elements are sorted, is

not specified.
30 23
14,52,62,79 25 88
98
31
30 14
23,31,52,88 25 62
98
79
253
Defining Measure of Progress
30 14
23,31,52,88 25 62
98
79
6 elements
254
Define Step
30 14
23,31,52,88 25 62
98
79
255
Define Step
• Select arbitrary element from side.
• Insert it where it belongs.
30 14
23,31,52,88 25 62
98
79
30 14
23,31,52,62,88 25
98
79
256
Making progress while Exit
Maintaining the loop invariant

79 km 75 km
30 14
23,31,52,88 25 62
98
79
30 14
6 elements 23,31,52,62,88 25
98
79
Sorted sub-list 5 elements
257
Beginning & km 0 km Exit Exit
Ending
88 52 14
31 25 62 30
98 23
79 88 52 14
31 25 62 30 n elements
98 23
79
14,23,25,30,31,52,62,79,88,98 0 elements
14,23,25,30,31,52,62,79,88,98
258
Running Time
Inserting an element into a list of size i
takes (i) time.
Total = 1+2+3+…+n =?
30 14
23,31,52,88 25 62
98
79
30 14
23,31,52,62,88 25
98
79
259
Adding Made Easy
1 + 2 + 3 + ... + n-1 + n = S = number of white dots
1 2........n
260
Adding Made Easy
n + n-1 + n-2 + ... + 2 + 1 = S = number of yellow dots
n ....... 2 1 1 2........n
261
Adding Made Easy
n + n-1 + n-2 + ... + 2 + 1 = S = number of yellow dots
n
2S dots
n
= n(n+1) dots in the
n
grid
n
n (n+1) n
S= 2 n+1 n+1 n+1 n+1 n+1
262
Adding Made Easy
263
Adding Made Easy
264
Running Time
Inserting an element into a list of size i
takes (i) time.
Total = 1+2+3+…+n = (n2)
30 14
23,31,52,88 25 62
98
79
30 14
23,31,52,62,88 25
98
79
265
Algorithm Definition Completed
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
266
Insertion Sort
as a Relay Race
88 52 14
31 25 62 30
98 23
79
267
Establish Loop Invariant
from preconditions
88 52 14
31 25 62 30
98 23
79
268
88 52 14
31 25 62 30
98 23
79
1
269
i-1
i
88 52 14
31 25 62 30
98 23
79
270
88 52 14
31 25 62 30
98 23
79
i
271
i-1
i
52 14
31,88 25 62 30
98 23
79
272
52 14
31,88 25 62 30
98 23
79
i
273
i-1
i
52 14
25,31,88 62 30
98 23
79
274
52 14
25,31,88 62 30
98 23
79
i
275
i-1
i
14
25,31,52,88 62 30
98 23
79
276
14,23,25,30,31,52,62,79,88,98
277
Clean up loose ends
1
T+
14,23,25,30,31,52,62,79,88,98
278
Code from LI - Sorting
You did not formally prove sorting works.
You did this handwavy picture thing.
From that how are we to know what to do?
Oh? Sorry.
I feel I did prove it formally.
I feel I followed the same steps.
But let’s try it again.
279
• Precond: A[1,..,n] are numbers.
• Postcond: A[1,..,n] is sorted.
• Loop Invariant: A[1,..,i] is sorted & A[i+1,..,n] are rest.
<loop-invariantt>
Exit
codeB
Let it and At be the values of i and A at the beginning of the

iteration and let it+1 and At+1 be their values after going around
again.
280
<loop-invariantt> At[1,..,it] is sorted & At[it+1,..,n] are rest
At[1,..,it] is sorted & At[it+1] & At[it+2,..,n] are rest
¬<exit Cond>
codeB: it+1 = it + 1
ng
rt
th i
Inse
What math do we
No
need from codeB?
At+1[1,..,it,it+1] is sorted & At+1[it+2,..,n] are rest

At+1[1,..,it+1] is sorted & At+1[it+1+1,..,n] are rest
<loop-invariantt+1>
281
Sum( A )
“preCond: A[1,..,n] are numbers”
?
loop
“loop-invariant:
A[1,..,i] is sorted & A[i+1,..,n] are rest.
exit when (i=n)
The inner loop is not what we
Insert A[i+1] into A[1,..,i] are working on now so it just
endloop can be written in English?
?
“postCond: A[1,..,n] is sorted”
282

<preCond>
<loop-invariant>
codeA
283
<Precond> A[1,..,n] are numbers

codeA: i=0
Give the LI with minimal

changes from the precond. Remember we love 0!
A[1,..,0] is sorted & A[1,..,n] are rest.

The empty set <loop-invariantt>
284
Clean up loose ends

<loop-invariant>
Exit
codeC
285
<loop-invariant> A[1,..,i] is sorted & A[i+1,..,n] are rest

The empty set
<exit Cond> i=n
codeC: Nothing
A[1,..,n] is sorted.
<postcond>
286
We said you don’t have
Sum( A ) to do the inner loop.
i=0
loop about the entire computation.
“loop-invariant:
exit when (i=n) one concept
at a time.
Insert A[i+1] into A[1,..,i]
endloop
nothing
“postCond: A[1,..,n]
the entireisalgorithm
sorted” at once
287
We said you don’t have
Sum( A ) to do the inner loop.
i=0
loop
“loop-invariant:
exit when (i=n)
Insert A[i+1] into A[1,..,i] But let’s do it for fun.
endloop
nothing
“postCond: A[1,..,n] is sorted”
288
Code from LI - Inserting
• Precond: a1 , ...,ai is sorted & c
• Postcond: a1,..,aj-1,c,aj,..,ai is sorted
• Loop Invariant: a1,..,aj-1,c,aj,..,ai with a’s sorted & c≤aj
• Partial Step: j = j-1
At[1,..,it] is sorted & At[it+1]
rt
Inse
What math do we
need from codeB?
At+1[1,..,it,it+1] is sorted
289
• Exit condition: aj-1<c
<loop-invariantt> a1,..,aj-2,aj-1,c,aj,..,ai with a’s sorted & c≤aj
¬<exit Cond> c ≤ aj-1

codeB: Swap
What math do we
need from codeB?
??
a1,..,aj-2,c,aj-1,aj,..,ai with a’s sorted & c≤aj-1
<loop-invariantt+1>
290
Insert(a1,...,ai & c)
“preCond: a1, ...,ai is sorted & c”
?
loop
“loop-invariant:
a1,..,aj-1,c,aj,..,ai with a’s sorted & c≤aj
exit when (aj-1<c)
j = j-1
endloop
?
“postCond: a1,..,aj-1,c,aj,..,ai is sorted”
291
<preCond>
<loop-invariant>
codeA
292
<Precond> a1 , ...,ai is sorted & c
codeA: Put c on end (if not already there)
& j=i+1
Give the LI with minimal

changes from the precond. There is not really a next
number to worry about
a1, ...,ai,c with a’s sorted & c≤ ?
<loop-invariantt>
293
Clean up loose ends

<loop-invariant>
Exit
codeC
294
<loop-invariant> a1,..,aj-1,c,aj,..,ai with a’s sorted & c≤aj
<exit Cond> aj-1<c

codeC: Nothing
Ensure consecutive
pairs are sorted.
a1,..,aj-1,c,aj,..,ai is sorted
<postcond>
295
Insert(a1,...,ai & c)
“preCond: a1, ...,ai is sorted & c”
j = i+1
loop
“loop-invariant:
a1,..,aj-1,c,aj,..,ai with a’s sorted & c≤aj
exit when (aj-1<c)
j = j-1
endloop
nothing
“postCond: a1,..,aj-1,c,aj,..,ai is sorted”
296
Ok
I know you knew Insertion Sort
But hopefully you are beginning to appreciate

Loop Invariants
for describing algorithms
297
Explaining Insertion Sort
We maintain a subset of elements sorted within a
list. The remaining elements are off to the side
somewhere. Initially, think of the first element in
the array as a sorted list of length one. One at a
time, we take one of the elements that is off to the
side and we insert it into the sorted list where it
belongs. This gives a sorted list that is one element
longer than it was before. When the last element
has been inserted, the array is completely sorted.
But don’t do this.

I want separate paragraphs
for the separate steps.
298
Insertion Sort
Selection
What changes?
This was on an exam and most got it
WRONG
• Loop Invariant:
• Select the smallest Action not picture
30 14
23,31,52,88 25 62
98
79
299
Insertion Sort
Selection
What changes?
This was on an exam and most got it
WRONG
• Loop Invariant:
• All left ≤ All right
31 88
14,23,25,30 ≤ 52 62
98
79
300
Define Step
31 88
14,23,25,30 ≤ 52 62
98
79
301
Define Step
31 88 88
14,23,25,30 ≤ 52 62 14,23,25,30,31 ≤ 52 62
98 98
79 79
Select the smallest on right

and put it at the end of the left.
302
Exit
31 88 88
14,23,25,30 ≤ 52 62 14,23,25,30,31 ≤ 52 62
98 98
79 79
<loop-invariantt>
¬<exit Cond>
codeB
• Numbers on
left were sorted. • Numbers now on
• New number added to end
left are sorted.
came from right.
• All left ≤ All right <loop-invariantt+1>303
Exit
31 88 88
14,23,25,30 ≤ 52 62 14,23,25,30,31 ≤ 52 62
98 98
79 79
<loop-invariantt>
¬<exit Cond>
codeB
• Was All left ≤ All right
• New number is smallest • Now
from right
All left ≤ All right
Don't Redo Work 35
X
30 A
• Prefix Averages: 25
• Input: an array X[i] of n values 20
• Output: The average of each 15
prefix 10
A[i] = (X[1] + X[2] + … + X[i])/i 5
0
1 2 3 4 5 6 7
for i = 1 to n do
Which line works the hardest?
% compute A[i] # times it is executed
=1+2+3+….+n n
n
s=0 = O(n2) n
for j = 1 to i do The other lines effect this time n
s=s+
by a multiplicative constant n
X[j]
A[i] = s / i or less n
n n n n n
return A Time(n) = O(n2)

305
Don't Redo Work
• Prefix Averages:
• Input: an array X[i] of n values
• Output: The average of each
prefix
A[i] = (X[1] + X[2] + … + X[i])/i
i=0; s = 0; Establish LI
loop
% LI: A[1], … , A[i] computed
% LI: s = X[1] + X[2] + … + X[i]
exit when i = n
i = i+1 Make “progress”
s = s + X[i] Maintain second LI
A[i] = s / i Maintain first
return A Exit Cond + LI  Post Cond
306
Don't Redo Work
• Prefix Averages:
• Input: an array X[i] of n values
• Output: The average of each
prefix
A[i] = (X[1] + X[2] + … + X[i])/i
i=0; s = 0;
loop
% LI: A[1], … , A[i] computed Which line works the hardest?
% LI: s = X[1] + X[2] + … + X[i] # times it is executed
exit when i = n =n
i = i+1
s = s + X[i]
The other lines effect this time
A[i] = s / i by a multiplicative constant
return A or less
Time(n) = O(n)
(Linear is better than quadratic.)
307
308
A Starry Night
How did Van Gogh come up with his famous
painting, A Starry Night?
There's no easy answer.
Designing algorithms is an art form.
309
A Sequencevs of Actions
A Sequence of Assertions
Max( a,b,c )
“preCond: Input has 3 numbers.”
m=a
if( b>m )
m=b
endif
“assert: m is max in {a,b}”
if( c>m )
m=c
endif It is helpful to have
“assert: m is max in {a,b,c}” different ways of
return(m)
looking at it.
“postCond:
return max in {a,b,c}” 310
Designing Loop Invariants
Coming up with the loop invariant is the
hardest part of designing an algorithm.
It requires practice, perseverance, and insight.
Yet from it
the rest of the algorithm
follows easily
311
Use This Process
Don't come up with the loop invariant after
the fact to make me happy.
Use it to design your algorithm.
312
Don’t start coding
You must design a working algorithm first.
313
Exemplification:
Try solving the problem
on small input examples.
Rudich www.discretemath.com 314

Start with Small Steps
What basic steps might you follow to make
some kind of progress towards the answer?
Describe or draw a picture of what the data
structure might look like after a number of
these steps.
315
Picture from the Middle
Leap into the middle of the algorithm.
What would you like your data structure to

look like when you are half done?
316
The Work Completed
The loop invariant should state what work
has been completed towards solving the
problem and what works still needs to be
done.
79 km 75 km
317
Flow Smoothly
The loop invariant should flow smoothly from
the beginning to the end of the algorithm.
– At the beginning, it should follow easily from
the preconditions.
– It should progress in small natural steps.
– Once the exit condition has been met, the
postconditions should easily follow.
318
Ask for 100%
A good philosophy in life is to ask for 100%
of what you want,
but not to assume that you will get it.
– What would you like to be true in the middle of
your computation?
319
Ask for 100%
Pretend that a genie has granted your wish.
– You are now in the middle of your computation
and your dream loop invariant is true.
320
Ask for 100%
Maintain the Loop Invariant:
– From here, are you able to take some
computational steps that will make progress
while maintaining the loop invariant?
321
Ask for 100%
• If you can maintain the loop invariant, great.
• If not,
– Too Weak: If your loop invariant is too weak, then the
genie has not provided you with everything you need to
move on.
– Too Strong: If your loop invariant is too strong, then you
will not be able to establish it initially or maintain it.
322
No Assumptions:
Suppose a Martian jumped
into the top of the loop.
All you would know is that <loop-invariant> was true.
It alone must provide enough information
so that, after going around the loop, you can establish
that the loop invariant is again true.
323
Know What a LI Is Not
``the LI is ...''
– code
– A precondition
– A postcondition
– A statement that is always true
Eg: “1+1=2”
– The steps taken by the algorithm
324
Differentiating between Iterations
x=x+2
– Meaningful as code
– False as a mathematical statement
x’ = xi = value at the beginning of the iteration
x” = xi+1 = new value after going around the
loop one more time.
x” = x’+2
– Meaningful as a mathematical statement
325
Max( A )
of n values.”
i=1
m = A[1]
loop
“loop-invariant:
i=i+1 codeB
return(m) in {A[1]..A[i]} in {A[1]..A[i+1]}
“postCond: m = max(m,A[i+1])
m is max in {A[1]..A[n]}” 326
<preCond> <exit Cond>  <postCond>
codeA
loop
<loop-invariant>
exit when <exit Cond> Same Miracle
codeB
codeC
<postCond> Miracle
Clean up loose ends

<loop-invariant>
Exit
codeC
Trivially true because

<exit Cond> = <postCond>
NEVER DO THIS! 327
<preCond> <exit Cond>  <postCond>
codeA
loop
<loop-invariant>
exit when <exit Cond> Same Miracle
codeB
codeC
<postCond> Miracle
The difficulty becomes

proving that it exits.
328
Find Errors
Which of the steps

to develop an iterative
algorithm did the student
fail to do correctly?
329
Find Errors
If I = -5, s = 1-5 j = 0
Fine: Describes input

and output.
330
Find Errors
Variables need to
be documented.
Fine: Purpose outlined
in LI
331
Find Errors
Loop Invariants are pictures

of current state.
Not actions!
Not algorithms!
332
Find Errors
Fine: Shows a relationship

between the variables.
333
Find Errors
Let s’ and i’ be
values of s and i
when at the top of the loop.
Let s” and i” be
values after going
around again.
334
Find Errors
LI  s’ = j=1i’ j.
Code  s” = s’+i’
i” = i’+1.
Together:
s” = (j=1i’ j) + i’.
i’ is added in twice.
i” should be added.
335
Find Errors
i = 1  j=1i j = 1 = s.
336
Find Errors
Exit
Better to say:
loop
exit when i>I
loop-invariant: _____
exit when i > I
337
Find Errors
Exit
LI  s = j=1i j.
exit when i>I
Exit  i > I
Together:
s = j=1I+1 j.
 Post Cond.
338
Find Errors
Test:
• on typical value
i = 3  s=1+2+3.
• on special values
exit when i>I
i = 1  s=1.
i = 0  s=1.
339
Typical Types of Loop Invariants
• Fairy God Mother
• More of the input
• More of the output
• Narrowing the search space
• Shrinking Input Instance
• Data Structure (System) Invariant
340
– I have arrived from Mars, some amount of
progress has been made already and my Fairy
God Mother has set the state of the computation
to be just what I want so that I feel as
comfortable as I could be given the amount of
progress that has been made.
• Data Structure (System) Invariant 341
Fairy God Mother
I run infinitely faster than the monster.

so my only goal is to get to
shore without him there.
The only problem is that

he runs 4 times faster than I swim.
Me: Radius 1, speed 1, time 1.
Him: Half circ., speed 4, time = 3.14/4 < 1
He wins.
342
Fairy
• Precond: I’m in somewhere in the lake. God Mother
He runs 4 I swim.
• Postcond: Get to shore without him there.
He runs back and forth to catch me.

And I change my direction to avoid him.
Can I get away?
343
79 km • Measure of Progress: Fairy God Mother
My distance from center
• Loop Invariant:
He is opposite of me.
My progress is the distance

swum from the center.
What is your wish?
It is my job to
• establish
• maintain
• and get the post condition
from this loop invariant
344
Establishing Loop Invariant Fairy God Mother
<preCond> I’m somewhere
codeA Swim to middle He is opposite of me
<loop-invariant>
I don’t want to panic and drown

by trying to solve the whole problem.
My goal now is not to escape or to make progress.
My only goal is to establish the loop invariant by ..?
Moving to the center.
Your motto should be:

0 is a fine number and is a fine  set.
345
Maintain Loop Inv
<loop-invt>
Fairy God Mother
Exit
He is opposite of me
¬<exit> ¬ (I can’t maintain LI)
codeB Swim opp direc He is opposite of me
& outward. <loop-invt+1>
My only goal is to maintain the loop invariant by ..?
by moving opposite him in my circle.

But he moves 4 times faster!!!
How long can I keep doing this?
And I must make progress When should I exit my “loop”?
by swimming out. When I can no longer maintain the LI. 346
Fairy God Mother
Similar Triangles
1 r rd His distance =d
d My distance = rd
His time = d/4
My Time = rd
Need rd < d/4
r < 1/4
So the circumference of my circle
must be ¼ of his. 347
Fairy God Mother
If I am less than ¼ from the middle,

then I can swim my circle faster
than he can run his.
(Even though he runs 4 times faster.)
Hence, I can maintain the loop invariant,

by moving opposite him
And can use my extra speed

to make progress outward.
348
Maintain Loop Inv
<loop-invt>
Fairy God Mother
Exit
¬<exit> I’m at < ¼
codeB Swim opp direc He is opposite of me
& outward. <loop-invt+1>
My only goal is to maintain the loop invariant by ..?
by moving opposite him in my circle.

I keep doing this as long as
the circumference of my circle < ¼ his
and then I will exit the “loop”.
349
Ending
<loop-inv>
Fairy God Mother
Exit <exit> I’m at = ¼ I am at the shore
codeC Swim for it! without him.
<post-cond>
When I am ¼ from the middle,
I can no longer maintain the loop invariant.
Hence, I … ?
Swim for it.
Me: ¾ radius, speed 1, time = ¾ = 0.75

Him: Half circ., speed 4, time = 3.14/4 = 0.78
I win!
350
Can’t Do Three In Row
Precondition: Infinite line of cells. Alternate white-black-white-….
PostConditions: Opponent can’t to 3 in a row.
b ww b b ww w b
Oops. Let’s always put

our b on the left first.
351
<loop-invariant> Can’t Do Three In Row
<loop-invariantt>
¬<exit Cond> Never stop
codeB: Aaah!
… ? b? w
x ? …
i
Assume we are at the top of the “loop”.

The loop invariants is true.
(or you can state that there is an error)
The opponent goes. Denote the cell’s index i.
We need to specify where we go.
And prove the loop invariant is true again.
There are a few cases.
353
a … x w … ? Impossible
i
b … b
b/x w
x b/x … ?
i
There weren’t blocks of 3 before
c … b ww b
x b/x … ? and we did not create a new one.
i
d … x w x b/x … ? Impossible
i
e … w w x b/x … ? Impossible
i
354
n in a Row
Precondition: Infinite line. Int n. Opponent skips every 10th move.
PostConditions: Continuous block of n whites.
Change slides to
2^3 = 8
or 2^4 = 16
Or I get 3 moves, he gets 1

(3/2)^5 =
9 * 2/3 = 6
6 * 2/3 = 4
4 * 2/3 =
3 * 2/3 = 3
355
n in a Row
b wwwww
wwwwww
Garbage that is ignored
wwwwww
wwwwww This “garbage” might help white
wwwwww because it does not have a black
on one side.
wwwwww
f(i) It might hurt black because he
wwwwww might waste his time putting a
wwwwww black on the other side.
wwwwww But it is too confusing for us to
keep track of.
wwwwww We want a clean picture/loop
wwwwww invariant.
So we ignore it. 356
n in a Row
wwwwww
wwwwww
wwwwww
wwwwww
wwwwww
f(i)
wwwwww
wwwwww
wwwwww
wwwwww
wwwwww
357
n in a Row
<loop-invariant>
<exit Cond> i=n , f(n)=1

codeC: Done
Actually, it is white’s turn, so
she could make it one loner,
but +1 might not be worth
the extra thought need.
f(n) w w w w w w w ww
=1 <PostConditions>
n 358
n in a Row
359
n in a Row
<loop-invariantt>
¬<exit Cond> i<n

codeB: We need to make each block one longer
n in a Row
wwwwwww b
wwwwww I think black will go here.
wwwwww
wwwwww But it can be really hard to know
wwwwww where black will go.
f(i) He could do a move that looks
wwwwww
stupid and then win because of it.
wwwwww How do you KNOW he does not
wwwwww have a better move?
wwwwww We have to assume he is SMART.
wwwwww
361
n in a Row
wwwwwww
wwwwww b wwwww b
wwwwww Garbage that is ignored
wwwwww
wwwwww Remember the garbage?
f(i) He might go there.
wwwwww
But this would be good for white
wwwwww because white has decided to
wwwwww ignore this block.
wwwwww Because white told black that it
will be ignored, we don’t have to
wwwwww
worry about black going there.
362
n in a Row
wwwwwww
wwwwww
Black could go here,
wwwwww b but it is a shorter block.
wwwwww
wwwwww
f(i)
wwwwww
wwwwww
wwwwww
wwwwww
wwwwww
363
n in a Row
b wwwwwww
wwwwww
wwwwww Black could go here,
but this is basically the same.
wwwwww
wwwwww The key is that black can kill
f(i) one of our “special” blocks.
wwwwww
wwwwww
wwwwww
wwwwww
wwwwww
364
n in a Row
wwwwwww b
wwwwwww b
This really does feel
wwwwwww b like black’s best move.
wwwwwww b
wwwwwww b Garbage
f(i)
wwwwwww b
wwwwwww b
wwwwwww b
wwwwwww b
wwwwwww f(i)/10 Black skip every 10th move.
365
n in a Row
Because f(i) goes down by a factor of 10 each iteration,

f(i) = c/10i
After n iterations, we need f(n)=1 block of length n.

f(n) = c/10n = 1
c =10n
f(i) = 10n/10i = 10n-i
i+1
wwwwwww f(i)/10 = f(i+1)
366
n in a Row
When, i=1, you need f(1) = 10n-1

single whites away from any black.
How are you going to do that!
My favorite number is i=0.

Then I need f(0) = 10n
blocks of length zero
away from any black.
367
n in a Row
f(0)
Done
368
n in a Row
w b
w b
This really does feel
w b like black’s best move.
w b
w b Garbage
f(0)
w b
w b
w b
w b Black skip every 10th move.
w f(0)/10 = f(1)
369
370
More of the Input
Loop Invariant
The input consists of an array of objects
Solution
Extra
We have read in the first i objects.

We will pretend that this prefix
is the entire input.
We have a solution for this prefix
plus some additional information
371
Loop Invariants and
Deterministic Finite Automaton
L = {a Î {0,1}* | a has length at most three and the
number of 1's is odd }.
372
More of the Input
Loop Invariant
Solution
Solution Extra
Extra
We read in the i+1st object.

We will pretend that this larger prefix
We extend the solution we have
to one for this larger prefix.
(plus some additional information) 373
More of the Input
Loop Invariant
Solution
Solution Extra
Extra
Exit
79 km 75 km
i to i+1
374
More of the Input
Loop Invariant
Solution
Exit
In the end, we have read in the entire input.
The LI gives us that we have a solution
for this entire input.
375
Insertion Sort
The input consists of an array of integers
52,23,88,31,25,30,98,62,14,79
Solution
23,31,52,88

We will pretend that this prefix
We have a solution for this prefix
376
Insertion Sort
The input consists of an array of integers
52,23,88,31,25,30,98,62,14,79
23,31,52,88
23,25,31,52,88
We read in the i+1st object.

We will pretend that this larger prefix
We extend the solution we have
to one for this larger prefix.
377
More of the Input
Loop Invariant
A student midterm answer:

Each iteration considers the next input item.
Loop Invariants are pictures
of current state.
Not actions!
Not algorithms!
378
More of the Input
Loop Invariant

We have considered the ith input element.
What about elements [1...i]?
379
More of the Input
Loop Invariant

We have considered input elements [1..i].
So?
380
More of the Input
Loop Invariant
Solution
We have a solution for each.
Each object does not need a separate solution

The input as a whole needs a solution.
381
More of the Input
Loop Invariant
Extra
Solution
We have a solution for the
prefix consisting of elements [1..i].
(plus some additional information)
382
Longest Block of Ones
00101111001100011111000011001
Alg reads the digits one at a time

and remembers enough about what read
so that it does not need to reread anything.
(i.e. a DFA)
383
00101111001100011111
When it has read this much,

what does it remember?
• Largest block so far.
Read the next character &

re-determine the largest block so far.
384
00101111001100011111
When it has read this much,

what does it remember?
• Largest block so far.
• Size of current block.
Read the next character &

re-determine the largest block so far
& current largest.
385
386
More of the Output
Loop Invariant
The output consists of an array of objects
I have produced the first i objects.
Produce the i+1st output object.
Exit
i to i+1 Exit
Done when
output n objects.
79 km 75 km
387
Selection Sort
Input:
52,23,88,31,25,30,98,62,14,79
The output consists of an array of integers
14,23,25,30,31,52,62,79,88,98
I have produced the first i objects.
Produce the i+1st output object.
Done when
Exit
output n objects.
388
More of the Input/Output
Loop Invariant
The input consists of a set of objects
Some i of the inputs have been handled.

We still have the precondition
on the input objects not yet handled.
The output consists of a set of objects
We have produced some i of the output.

We have the post condition
on the output objects produced.
389
Leaning Tower
Input: Lots of 2in wide blocks
Output: Stack them leaning as far over as possible.
With out falling
How far over can you shift it go over?
More of the output?
390
Leaning Tower
But if I keep adding blocks

it will falling
Is this a proof that

it is not possible?
Try again 
More of the output:
I have produced the first i-1 objects.
391
Leaning Tower
it will falling
Where should we
put the next block?
More of the output:

I have produced the last i-1 objects.
Extra info: Center of mass is within bottom block
Leaning distance so far 392
Leaning Tower
it will falling
Where should we
put the next block?
More of the output:

Leaning Tower
Where should we No progress

put the next block?
More of the output:

Leaning Tower
With the center of mass

Where should we of the above blocks
put the next block? teetering on the edge
of supporting block.
More of the output:
Leaning Tower
Where is the new

center of mass?
More of the output:

Leaning Tower
Mass = m1 Mass = m2
Where is the new

center of mass?
Center of mass = r1
Center of mass = r2
Center of mass
m1r1  m2 r2
R
m1  m2
(Ignoring veridical)
397
Leaning Tower
mass2
mass1
To make the math easier,

Where is the new
Let's call this zero.
center of mass?
Center of mass = r1 = 1 Mass = m1 = 1
Center of mass = r2 = 0 Mass = m2 = i-1
Center of mass
R
m1r1  m2 r2 Center of mass is 1/i from
m1  m2
11  (i  1)  0 1 right edge of ith block.
 
1  (i  1) i 398
Leaning Tower
1
1/i
Add the next block. d  i 1.. n 1  ln( n)
i
n  ed
More of the output:

Extra info: Center of mass is 1/i from
right edge of ith block.
Leaning distance so far? 399
Leaning Tower
1
1/i
d  i 1.. n 1  ln( n)
i
n  2d
https://www.youtube.com/watch?
v=pBYPXsGka74&feature=youtu.be 400
Leaning Tower
https://www.youtube.com/watch?
v=pBYPXsGka74&feature=youtu.be 401
König’s Lemma
König’s Lemma: Let T be a rooted tree with an infinite number of nodes.
It either has
• a node with infinite degree
• or a path of infinite length.
∞
∞
402
König’s Lemma
König’s Lemma: Let T be a tree with infinite nodes ∞ i
It either has a node with infinite degree
or a path of infinite length. ∞
Proof: We will build an infinite path one node at a time.

loop S(i) ≡ “We have either found a node with infinite degree
inv: or a path of length i to a node
whose subtree contains and infinite number of nodes.”
403
König’s Lemma
König’s Lemma: Let T be a tree with infinite nodes
or a path of infinite length.
Proving S(0): Start at the root, with a path of length 0.
404
König’s Lemma
König’s Lemma: Let T be a tree with infinite nodes ∞
or a path of infinite length.
Assume S(i) and proving S(i+1):
If we have found a node with infinite degree already
or the current node has infinite degree we are done.
405
König’s Lemma
König’s Lemma: Let T be a tree with infinite nodes i
It either has a node with infinite degree r

Else we are at node with finite degree
whose subtree has an infinite number of nodes
406
König’s Lemma
König’s Lemma: Let T be a tree with infinite nodes i
It either has a node with infinite degree r
or a path of infinite length. n1 n2 n∞3 nr

Let n and n1, n2, n3, …, nr, denote the number of nodes in the subtrees
of our current node and its children.
Note n = n1 + n2 + n3 + … + nr + 1.
By our assumptions n is infinite and r is finite.
Hence one of the children must have infinite nj.
407
König’s Lemma
König’s Lemma: Let T be a tree with infinite nodes
It either has a node with infinite degree i+1

Let n and n1, n2, n3, …, nr, denote the number of nodes in the subtrees
of our current node and its children.
Note n = n1 + n2 + n3 + … + nr + 1.
By our assumptions n is infinite and r is finite.
Hence one of the children must have infinite nj.
Move to a child, giving S(i+1). This completes the proof.
408
• More of the output Three
• Narrowing the search space Binary Search like
• Shrinking Input Instance examples
409
• More of the output Three
• Narrowing the search space Binary Search like
• Shrinking Input Instance examples
410
Define Problem: Binary Search
• PreConditions
– Key 25
– Sorted List
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
• PostConditions
– Find key in list (if there).
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
411
• Maintain a sub-list
• Such that
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
412
• Maintain a sub-list.
• If the key is contained in the original list,
then the key is contained in the sub-list.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
413
Define Step
• Make Progress
• Maintain Loop Invariant
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
414
Define Step
• Cut sub-list in half.
• Determine which half the key would be in.
• Keep that half.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
415
Define Step
• Keep that half.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key ≤ mid, If key > mid,

then key is in then key is in
left half. right half. 416
Define Step
• It is faster not to check if the middle
element is the key.
• Simply continue.
key 43
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95

Exit
Make Progress 79 km 75 km
• The size of the list becomes smaller.
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
79 km
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
75 km
418
km
Initial Conditions
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
n km
• If the key is contained in

• The sub-list is the the original list,
entire original list. then the key is contained
in the sub-list.
419
Ending Algorithm Exit
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
0 km
• If the key is contained in • If the key is

the original list, contained in the
then the key is contained original list,
in the sub-list. then the key is at
• Sub-list contains one this location.
element. Exit
Check this location.
• If it’s the key, then you have found it.
• If it’s not, then
it was not in the original list. 420
Running Time
The sub-list is of size n, n/2, n/4, n/8,…,1
Each step (1) time.
Total = (log n)
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95

Code
422
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
423
Study:
• Many experienced programmers were asked
to code up binary search.
80% got it wrong

Good thing is was not for a
nuclear power plant.
424
Make Precise Definitions
• Maintain a sub-list with end points i & j.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
i j
425
• Maintain a sub-list with end points i & j.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
i j
Does not matter which,
but you need to be consistent.
426
• If the sub-list has even length,
which element is taken to be mid?
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
i  j  i  j 
mid    or mid    ?
 2   2 
427
• If the sub-list has even length,
which element is taken to be mid?
mid
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
Should not matter i  j 

mid  
 2 
Choose right.
428
Common Bugs
• Keep that half.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95

Common Bugs
• If the middle element is the key, Exit
it can be skipped over.
key 43
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95

Common Bugs
• Fix the bug.
key 43
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95

Common Bugs
• Second fix,
by making the left half slightly bigger.
• New bug?
key 43
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95

Exit
Common Bugs 79 km 75 km
• Second fix,
by making the left half slightly bigger.
• New bug?
key 43
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
No progress is made.
Loop for ever! 433
Why is Binary Search so Easy to Get Wrong?
• How many
i  j  i  j 
1 mid    OR   ? possible
 2   2  algorithms?
• How many
2 if key  L(mid) OR > ?
correct
i = mid i = mid + 1 algorithms?
3 • Probability of
else OR else
success by
guessing?
j = mid -1 j = mid
434
Moral
• Use the loop invariant method to think
about algorithms.
• Be careful with your definitions.
• Be sure that the loop invariant is always
maintained.
• Be sure progress is always made.
435
Loop Invariants
for
A second
Binary Search like
example
436
Binary Search Tree
• <preCond>
– Key 17
– A binary search tree.
38
25 51
key 17?
17 31 42 63
4 21 28 35 40 49 55 71
• <postCond>
– Find key in BST (if there).
437
Binary Search Tree
Data Structure Invariant
Its left children ≤ Any node ≤ Its right children
38
25 51
key 17? ≤ ≤
17 31 42 63
4 21 28 35 40 49 55 71
438
Binary Search Tree
Search Loop Invariant
• We have maintained a sub-tree.
• If the key is contained in the original tree,
then the key is contained in this sub-tree.
38
25 51
key 17? ≤ ≤
17 31 42 63
4 21 28 35 40 49 55 71
439
Binary Search Tree
• Cut sub-tree in half.
• Keep that half.
38
25 51
key 17? ≤ ≤
17 31 42 63
4 21 28 35 40 49 55 71
If key < root, If key = root, If key > root,

then key is then key is then key is
in left half. found in right half.
440
Binary Search Tree
Search Loop Invariant
• We have maintained a sub-tree.
• If the key is contained in the original tree,
then the key is contained in this sub-tree.
38
25 51
key 17?
17 31 42 63
4 21 28 35 40 49 55 71
441
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
442
Loop Invariants
for
A third
Binary Search like
example
443
Card Trick
• A volunteer, please.
444
Pick a Card
Done
445
Loop Invariant:
The selected card is one of
these.
446
Which column?
left
447
Loop Invariant:
these.
448
Selected column is placed
in the middle.
449
I will rearrange the cards.
450
Relax Loop Invariant:
I will remember the same about
each column.
451
Which column?
right
452
Loop Invariant:
these.
453
in the middle.
454
I will rearrange the cards.
455
Which column?
left
456
Loop Invariant:
these.
457
in the middle.
458
Here is your card.
Wow!
459
The “Partitioning” Problem
(used in quicksort)
Input: Output:
p=52
88 52 14 88
14 98
31
25 98 30 31 30 23 ≤ 52 ≤ 62
25 79
62 23
79
460
p=52
31 23 25 98 30 14 62 79 88
≤p p≤
or
p=52
31 23 71 25 98 30 62 79 88
≤p p≤
461
Defining Measure of Progress
p=52
31 23 25 98 30 14 62 79 88
4 elements
not
looked at
462
Define Step
• Make Progress
p=52
31 23 25 98 30 14 62 79 88
≤p p≤
463
Define Step
• Make Progress
p=52
31 23 25 98 30 14 62 79 88
≤p p≤
p=52
31 23 14 25 98 30 62 79 88
≤p p≤
464
Define Step
Four cases
p=52 p=52
31 23 25 98 30 14 62 79 88 31 23 14 25 98 30 62 79 88
31 23 14 25 98 30 62 79 88 31 23 14 25 98 30 62 79 88
p=52 p=52
31 23 25 98 30 71 62 79 88 31 23 71 25 98 30 62 79 88
31 23 25 98 30 71 62 79 88 31 23 25 98 30 71 62 79 88
465
Beginning & km 0 km Exit Exit
Ending
p=52
88 52 14
31 25 62 30
98 23 p=52
79
88 25 31 98 62 14 30 79 23
n-1 elements
not
looked at
p=52
23 30 25 31 14 62 98 79 88
0 elements
25 62
not 79
looked at 23 31 14 ≤ 52 ≤ 98
30 88
466
Running Time
Each iteration takes (1) time.
Total = (n)
p=52
31 23 25 98 30 14 62 79 88
≤p p≤
p=52
31 23 14 25 98 30 62 79 88
≤p p≤
467
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
468
31 88 25 52 98 62 14 30 79 23
p=52 88 25 31 98 62 14 30 79 23
23 88 25 31 98 62 14 30 79
23 25 31 98 62 14 30 79 88
23 25 31 98 62 14 30 79 88
Tracing an
23 30 25 31 98 62 14 79 88
Example
23 30 25 31 98 62 14 79 88
23 30 25 31 98 62 14 79 88
23 30 25 31 62 14 98 79 88
23 30 25 31 14 62 98 79 88
23 30 25 31 14 62 98 79 88
469
An instance has been produced that is smaller
than the actual instance and these two instances
require the same output. Hence, it is suﬃcient for
me to forget about the actual instance and simply
give the output for the new smaller one.
470
GCD(a,b) Greatest Common Divisors
Input: <a,b> = <64,44>
Output: GCD(a,b) = 4
<a,b>  <a-b,b>
GCD(a,b) = GCD(a-b,b)
GCD(64,44) = GCD(20,44) = 4
Maintain values <x,y> such that
GCD(a,b) = GCD(x,y)
Let xt and yt be the values of x and y at the
beginning of the iteration and let xt+1 and
yt+1 be their value after going around again.
471
Input: <a,b> = <64,44>
<a,b>  <a-b,b>
GCD(64,44) = GCD(20,44) = 4
GCD(a,b) = GCD(x,y)
<loop-invariant
¬<exit Cond> t> GCD(a,b) = GCD(xt,yt)
codeB: xt+1 = xt-yt , yt+1 = yt
GCD(a,b) = GCD(xt,yt) = GCD(xt-yt,yt) = GCD(xt+1,yt+1)
<loop-invariant >472
Input: <a,b> = <64,44>
<a,b>  <a-b,b>
GCD(64,44) = GCD(20,44) = 4
GCD(a,b) = GCD(x,y)
Exit
79 km 75 km
codeB: xt+1 = xt-yt
x is smaller
473
GCD(a,b)
Input: <a,b> = <64,44>
<x,y> = <64,44>
= <20,44> GCD(a,b) = GCD(x,y)
= <44,20>
= <24,20>
= < 4,20>
= <20, 4>
= <16, 4>
= <12, 4>
Running time?
= < 8, 4>
= < 4, 4>
= < 0, 4>
GCD(a,b) = GCD(x,y) = 4 474
GCD(a,b)
Input: <a,b> = <9999999999999,2>
<x,y> = <9999999999999,2>
= <9999999999997,2>
= <9999999999995,2>
= <9999999999993,2>
= <9999999999991,2>
Time = O(a)
Poly Time?
475
Running Time
Grade School vs Kindergarten
 ********
********
a × b = a + a + a + ... + a
********
********
********
n2 ********
********
b
********
********
********
****************
T(n) = Time multiply T(n) = Time multiply

= θ(n2) = quadratic time. = θ(b) = linear time.
Which is faster?
92834765225674897 × 838839775901103948759
Which would you use? Minutes? Add a digit?
Centuries? Exponential? 476
The Time Complexity of an Algorithm
• Specifies how the running time
depends on the size of the input.
A function mapping Work for me to

give you the instance.
Work for you to
“size” of input read the instance.
“time” T(n) executed . Work for you to

solve it.
477
Size of Input Instance
83920
478
5
83920 1’’
2’’
• Size of - n = 2 in2
paper - n = 17 bits
Which are reasonable?
• # of bits - n = 5 digits
• # of digits - n = 83920
• Value
479
5
83920 1’’
2’’
• Size of - n = 2 in2 • Intuitive
paper - n = 17 bits
• # of digits - n = 83920
• Value
480
5
83920 1’’
2’’
paper - n = 17 bits • Formal
• # of digits - n = 83920
• Value
481
5
83920 1’’
2’’
• # of bits • Reasonable
- n = 5 digits
• # of digits - n = 83920 # of bits =
3.32 * # of digits
• Value
482
5
83920 1’’
2’’
- n = 5 digits
• Unreasonable
• # of digits - n = 83920
• Value # of bits = log2(Value)
Value = 2# of bits
483
• depends on the size of the input.
A function mapping
Work for me to
Work for you to

solve it.
484
Running Time
 ********
********
a × b = a + a + a + ... + a
********
********
********
n2 ********
********
b
********
********
********
****************

Which is faster?
92834765225674897 × 8388397759011039475
n = # digits = 20 b = value = 8388397759011039475
Time ≈ 202 ≈ 400 Time ≈ 8388397759011039475 485
Running Time
 ********
********
a × b = a + a + a + ... + a
********
********
********
n2 ********
********
b
********
********
********
****************

Which is faster?
92834765225674897 × 8388397759011039475
n = # digits = 20 b = value ≈ 10n
Time ≈ 202 ≈ 400 Time ≈ 10n ≈ exponential!!! 486
Running Time
 ********
********
a × b = a + a + a + ... + a
********
********
********
n2 ********
********
b
********
********
********
****************

Which is faster?
92834765225674897 × 83883977590110394759
n = # digits = 20 Adding a single digit
Time ≈ 202 ≈ 400 multiplies the time by 10!487
Grade School Addition: Linear time
Grade School Multiplication: Quadratic time
Kindergarten Multiplication: Exponential time
Fast Fourier Transform: O(n logn) time
t
i
m
e
input size
For big inputs, n << n2 << 2n
Oops this was worse!

488
Cryptography
n = # of bits vs N = value
Confused. Let's try again.

By considering
on line banking.
489
Cryptography
I set up my banking password as follows.

• I randomly choose two n digit primes
wolframalpha:
next prime 90843823823904
gives p = 90843823823947
next prime 38475756939837
gives q = 38475756939857
490
Cryptography
I set up my banking password as follows.

• I randomly choose two n digit primes
n= 14
p = 90843823823947 and q = 38475756939857
• I multiply N = pq = 3495284884937375456635355579
Time = O(n2)  142 = 196.
• I make this product N public.
• The pq factoring is my password.
491
Cryptography
Linear in the value N
not the number of digits n
• I know his public product n= 28

N = pq = 3495284884937375456635355579
• I simply factor to find pq.
Then I can steal his money.
loop p = 2..N
check if N/p
Time = O(N) = O(1028) = O(10n) = exponential time.
No problem this is linear time.
492
Cryptography
Linear in the value N
not the number of digits n
Time complexity is a function
Work for me to
give you the instance. n= 28
n= 28
Work for you to N = 3495284884937375456635355579
solve it. Time = 1028
>> Age of universe 493
Is this Silly?
I know other professors
would give this answer.
But I insist on a different

Silly(N) answer!
loop i = 1..N
Print( “HI”)
Time = O(N) = 2O(n)
Size = n = O(log(N))
N = 2O(n)
494
Is this Silly?
If N is one 32-bit integer,
then N≤231
Silly(N)
loop i = 1..N
Print( “HI”)
Time = O(N) = 2O(n) = O(1)
Size = n = O(log(N))
N = 2O(n)
495
Is this Silly?
If N is stored in n
32-bit integers
Silly(x0,x1,x2,…, xn-1) Silly(x0,x1,x2,…, xn-1)

Silly(N)
N = Σi=0..n-1 xi (232)i
loop
loop ii == 1..N
1..N loop i = 1..n
Print(
Print( “HI”)
“HI”) Print( “HI”)
Time = O(N) Time = O(n)
Size = n = O(log(N)) Size = n
N = 2O(n)
Clearly Clearly
Exponential! Linear! 496
• depends on the size of the input.
A function mapping
Work for me to
Work for you to

solve it.
497
Size of Input N,M x1,x2,…, xN Time

for xy
log2N +log2M N log2x (log2x)2 bit ops
# bits formal n formal hastel formal O(n2)
log10N +log10M N log10x (log10x)2 char ops
# digits same n same hastel same O(n2)
2 N 1 value op
# values too small standard n standard O(1)
N
values too big
498
Size of Input N,M x1,x2,…, xN Time

for xy
log2N +log2M (log2x)2 bit ops
# bits formal n formal O(n2)
N 1 value op
# values standard n standard O(1)
499
GCD(a,b)
Input: <a,b> = <9999999999999,2>
<x,y> = <9999999999999,2>
= <9999999999997,2>
= <9999999999995,2>
= <9999999999993,2>
= <9999999999991,2>
Work for me to
give you the instance. n= 13
Work for you to n= 13

solve it. a = 9999999999999
500
GCD(a,b)
Input: <a,b> = <9999999999999,2>
<x,y> = <9999999999999,2>
= <9999999999997,2>
= <9999999999995,2>
= <9999999999993,2>
= <9999999999991,2>
Time = O(a) = 2O(n)

Size = number of digits n = O(log(a))
501
GCD(a,b)
Speedup
<x,y>  <x-y,y>
 <x-2y,y>
 <x-3y,y>
 <x-4y,y>
 <x-iy,y>
 <remainder with x/y,y>
= <x mod y,y>
 <y,x mod y> smaller than y
502
GCD(a,b)
Input: <a,b> = <44,64>
<x,y> = <44,64>
= <64,44> GCD(a,b) = GCD(x,y)
= <44,20> <x,y>  <y,x mod y>
= <20, 4>
= < 4, 0>
GCD(a,b) = GCD(x,y) = 4
503
GCD(a,b)
Input: <a,b> = <10000000000001,9999999999999>
<x,y> = <10000000000001,9999999999999>
= <9999999999999,2>
Little progress
= <2,1> Lots of progress
= <1,0>
GCD(a,b) = GCD(x,y) = 1
Every two iterations:

the value x decreases by at least a factor of 2.
the size of x decreases by at least one bit
Time = O(log(a)+log(b)) = O(n)

Size = n = O(log(a)+log(b)) 504
GCD(a,b)
505
Multiplying
• Precondition: Integers X and Y.
• Postcondition: XY
• Elementary School Algorithm: XY = Y+Y+Y+…+Y

s=0 • What is the loop invariant?
for( i=1..X )
s = s+Y
• What relationship
return(s) do you want maintain
between the data items
X,Y,i,s?
• s=iY
506
Multiplying

s=0 • Great, but Let's try a Shrinking
for( i=1..X ) Input Instance Loop invariant.
s = s+Y – An instance has been
return(s) produced that is smaller than
the actual instance and these
two instances require the same
output.
507
Multiplying

s=0 • Great, but Let's try a Shrinking
x=X, y=Y Input Instance Loop invariant.
loop – An instance has been
exit when x=0 produced that is smaller than
x = x-1 the actual instance and these
s = s+Y two instances require the same
return(s) output.
• What is the maintained
x = X-i, y = Y relationship between X,Y,x,s?
s = iY = (X-x)Y = XY - xy • XY = xy + s
508
Multiplying
• Loop Invariant: XY = xy + s
Forget the algorithm.
Let's develop it using the steps.
509
Multiplying

<preCond>
<loop-invariant>
codeA
With the minimal work possible

x=X, y=Y
s=0
510
Multiplying
79 km • Measure of Progress
• x.
Exit
79 km 75 km
• x = x-1
• Exit Condition
Exit • x=0 511
Multiplying
Clean up loose ends

<loop-invariant>
Exit
codeC
512
Multiplying
<loop-invariant> XY = xy + s = s

<exit Cond> x=0
codeC: Return( s )
Return( XY ) <postCond> 513

Multiplying

Exit
<loop-invariantt>
codeB
Exit
79 km 75 km
• x = x-1
514
Multiplying
Exit
79 km 75 km
• x = x-1
515
Multiplying
Exit
79 km 75 km
• x = x-1
516
Multiplying
Exit

517
Multiplying

518
Multiplying

Exit
<loop-invariantt>
codeB
Let xt, yt, and st be the values of x, y, and s at the

beginning of the iteration and let xt+1, yt+1, and
st+1 be their value after going around again.
519
Multiplying
Exit
<loop-invariantt> Use the fact that you
¬<exit Cond> already have the LI true
codeB: and take a small step.
Make the loop Invariant true!!!
Solve the entire algorithm!!!
Panic!!
next
Multiplying
Exit
<loop-invariantt> XY = xtyt + st
¬<exit Cond> xt > 0
codeB: xt+1 = xt - 1 yt+1 = yt
st+1 = (st ?)
X Y
Write down LHS/RHS
of what we must prove
= xt+1yt+1 + st+1 <loop-invariantt+1>521
Multiplying
Exit
st+1 = (st ?)
XY = xtyt + st Use the LI
= (xt -1)yt + (st ?) Plug in new values.

Multiplying
Exit
st+1 = (st ?)
XY = xtyt + st
= xtyt – yt + (st ?) Rearrange
= (xt -1)yt + (st ?)
= x y +s <loop-invariantt+1>523
Multiplying
Exit
st+1 = st + yt
= xtyt – yt + (st + yt)
= (xt -1)yt + (st ?)
Multiplying
Exit
<loop-invariantt>
st st yt st+1
1
X xt
= = x -1
t =x t+1
Y yt yt yt
= xtyt – yt + (st + yt)
= (xt -1)yt + (st ?)
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
526
Multiplying

s=0
x=X, y=Y
loop
exit when x=0
x = x-1
s = s+y
return(s)
Time = O(X)
Linear Time?
527
5
83920 1’’
2’’
- n = 5 digits
• Unreasonable
• # of digits - n = 83920
• Value # of bits = log2(Value)
Value = 2# of bits
528
Multiplying

s=0
x=X, y=Y
loop
exit when x=0
x = x-1
s = s+y
return(s)
# iterations = O(X) = 2O(n)
Size = number of digits= nO(log(X))
529
Multiplying

s=0
x=X, y=Y
loop
exit when x=0
x = x-1 x/2
s = s+y
return(s)
We need to make X smaller faster!

530
Multiplying

Exit
<loop-invariantt>
codeB
Exit
79 km 75 km
• x = x/2
531
Multiplying
Exit
codeB: xt+1 = xt /2 yt+1 = (yt ?)
st+1 = (st ?)
= (xt /2)(yt ?)+ (st ?) Plug in new values.

Multiplying
Exit
codeB: xt+1 = xt /2 yt+1 = yt2
st+1 = (st ?)
= (xt /2) (yt2)+ (st ?)

Multiplying
Exit
codeB: xt+1 = xt /2 yt+1 = yt2
st+1 = st
= (xt /2) (yt2)+ st

Multiplying
Exit
<loop-invariantt>
st st st+1
X xt
= ½ =x /2
t =x t+1
Y yt yt yt+1
= (xt /2) (yt2)+ st

Multiplying
• Ethiopian Algorithm (Practice Test 1):
s=0
x=X, y=Y
loop
exit when x=0
if(x is even) “Lots” of progress because
x = x/2, y=2y x loses a bit!
else
x = x-1, s = s+y “Lots” of progress because
return(s) x becomes even!
# iterations = O(log(X)) = O(n)
Size = number of digits n = O(log(X)+log(Y)
536
Multiplying
s=0
x=X, y=Y
loop
exit when x=0
if(x is even) X < 2n, X < 2n,
x = x/2, y=2y XY < 22n,
else Sum Time = O(n)
x = x-1, s = s+y
Total Time = O(n2)
return(s)
# iterations = O(log(X)) = O(n)
Size = number of digits n = O(log(X)+log(Y)
537
Multiplying
538
Multiplying
• Allowed Operations of limited machine:

– Adding: a+b
– Doubling: a+a
– Comparisons: if( a<b ) then
• Not Allowed Operations:
– Bit operations: ith bit of x
– Divisions: x/2
539
Multiplying
s=0
x=X, y=Y
loop
exit when x=0
if(x is even)
x = x/2, y=2y x=x–u
else (Where u is as large
x = x-1, s = s+Y as possible.)
return(s)
540
Multiplying
Exit
codeB: xt+1 = xt - ut yt+1 = yt
st+1 = (st ?)
= (xt -ut)yt + (st ?) Plug in new values.

Multiplying
Exit
st+1 = (st ?)
= xtyt – utyt + (st ?) Rearrange
= (xt -ut)yt + (st ?)
Multiplying
Exit
st+1 = st + vt, vt = utY
= xtyt – utyt + (st+utY)
= (xt -ut)yt + (st ?)
Multiplying
Exit
<loop-invariantt>
st st u t yt st+1
X xt ut
= = x -u
t t =x t+1
Y yt yt yt
= xtyt – utyt +
= (xt -ut)yt + (st ?)
Multiplying
• Faster Algorithm:
s=0
x=X, y=Y
loop
exit when x=0
x = x-u[i]
s = s+v[i] with v[i] = u[i]Y
return(s)
Where u[i] is as large as possible.

How can we produce these u[i] and v[i] = u[i]Y?
545
Multiplying
• Allowed Operations of limited machine:

– Adding: a+b
– Doubling: a+a
– Comparisons: if( a<b ) then
• Not Allowed Operations:
– Bit operations: ith bit of x
– Divisions: x/2
Where u[i] is as large as possible.

How can we produce these u[i] and v[i] = u[i]Y?
546
Multiplying
Producing u[i] and v[i] with v[i] = u[i]Y:
i=0
u[i] = 1
v[i] = Y
loop
exit when u[i]>x u[i] will be subtracted from X.
i = i+1
u[i] = u[i-1]+u[i-1]
v[i] = v[i-1]+v[i-1]
• What is the loop invariant?

– u[i] = 2i
– v[i] = 2iY
547
Multiplying
u[i] = 2i, v[i] = 2iY, i = 0..?
s=0
x=X, y=Y
i = the last value in producing u[i]
loop
exit when x=0
while( x < u[i] ) i=i-1
x = x-u[i] But we don’t want x
to become negative.
s = s+v[i]
return(s)
548
Multiplying
u[i] = 2i, v[i] = 2iY, i = 0..?
s=0
x=X, y=Y
i = the last value in producing u[i]
loop
exit when x=0
while( x < u[i] ) i=i-1
x = x-u[i] Actually we will see that
s = s+v[i] u[i] gets used once if the ith bit of x is 1
return(s) and zero times if its is 0.
549
Multiplying
x3 = 1102
x = 10101102
u[0] = 12
u[1] = 102
u[2] = 1002
u[7] = 100000002
Actually we will see that

u[i] gets used once if the ith bit of x is 1
and zero times if its is 0.
550
Multiplying
Loop Invariants given in Practice Test:
LI0: u[i] = 2i, v[i] = 2iY, i = 0..?
LI1: X × Y = x × Y + s
LI2: x ≥ 0
LI3: x < 2i = u[i]
Progress: i = i-1

Exit
<loop-invariantt>
codeB
Exit
79 km 75 km
• i = i-1
551
Multiplying
Exit
79 km 75 km
• i = i-1
552
Multiplying
Exit
79 km 75 km
• i = i-1
553
Multiplying
Exit

554
Multiplying

555
Multiplying
LI0: u[i] = 2i, v[i] = 2iY, i = 0..?
LI1: X × Y = x × Y + s Exit
LI2: x ≥ 0
LI3: x < 2i = u[i]

Exit
<loop-invariantt>
codeB
Let xt, st, and it be the values of x, s, and i at the
beginning of the iteration and let xt+1, st+1, and
it+1 be their value after going around again.
556
Multiplying
LI0: u[i] = 2i, v[i] = 2iY, i = 0..?
LI1: X × Y = x × Y + s Exit
LI2: x ≥ 0
LI3: x < 2i = u[i]
There are two case:
By case xt < u[it-1] = u[it+1]

No change needed.
xt+1 = xt <u[it+1]
because LI3 is already true.
557
Multiplying
LI0: u[i] = 2i, v[i] = 2iY, i = 0..?
LI1: X × Y = x × Y + s
LI2: x ≥ 0
LI3: x < 2i = u[i]
There are two case:
By case xt > u[it-1] = u[it+1]

We need to make xt smaller to reestablishes LI3.
0 < xt+1 = xt - u[it-1] < u[it-1] = u[it+1]
This reestablishes LI3.
558
Multiplying
LI0: u[i] = 2i, v[i] = 2iY, i = 0..?
LI1: X × Y = x × Y + s
LI2: x ≥ 0
LI3: x < 2i = u[i]
There are two case:
xt+1 = xt - u[it-1]
st+1 = st + v[it-1]
559
Multiplying
u[i] = 2i, v[i] = 2iY, i = 0..?
s=0
x=X, y=Y
i=
loop the last value in producing u[i]
exit when x=0
i = i-1
if( x > u[i] )
x = x-u[i]
s = s+v[i]
return(s) 560
561
Data Structure Invariants
How about invariants for
a data structure or a system?
The importance of invariants is the same.

Differences:
1. An algorithm must terminate with an answer,
while systems and data structures may run
forever.
2. An algorithm gets its full input at the beginning,
while data structures gets a continuous stream
of instructions from the user.
• Both have invariants that must be maintained.
562
An Abstract (Public) Data Structure:
The minimal needed by the User so he can
• understand how best to visualize the data User
• and what each operation does.
Eg of Public Invariants:
• Bank: The amount is not negative.
• Stack: Contains an ordered list of objects which cannot
be changed except at the top via the explicit push and
pop operations. Important in recursion and parsing.
• Drug Allocation Data Base: A patient is not
simultaneously prescribed a set of drugs that interact
poorly with each other.
The implementer must make sure that each operation
maintains these invariants. Implementer
563
An Implemented (Private) Data Structure:
• The details need to be worked out so that operations
are correct and efficient. User
• This is all hidden from the User.
Eg of Private Invariants:
• Bank: Charge extra fees.
• Stack:
or
• Drug Allocation Data Base:
For each patient give name address
and list of drugs.
Implementer
564
Constructor
preCondConstructor InvariantsData Struc
When a user wants to construct an instance

of data structure the implementer must
make sure that its constructor establishes
all of its loop-invariants.
Initially it contains no objects.
565
Clean up loose ends
InvariantsData Struc
Exit Destructor postCondData Struc
preCondDestructor
A destructor for the data structure
must clean up loose ends.
566
Exit
InvariantsData Struc t InvariantsData Struc t+1
Push Operation
preCondPush postCondPush
Assume we fly in from Mars and InvariantsData Struc t is true:
Assume the user correctly calls the Push Operation:

preCondPush The input is info for a new element.
Implementer must ensure:
postCondPush The element is pushed on top of the stack.
InvariantsData Struc t+1
567
InvariantsData Struc t
preCondPush
next
postCondPush
568
preCondPush
Don’t panic.
Just draw the pictures
and move the pointers.
postCondPush
569
preCondPush
postCondPush
570
InvariantsData Struc t Special Case: Empty
preCondPush
postCondPush
571
preCondFind
Suppose you want to
find where the new
element 6 would go?
How do you get there?

Walk!
postCondFind
We want the location sandwiched.
572
preCondFind
What would you like to

be true in the middle
of your computation?
postCondFind
573
How do you
initialize this walk?
Sandwich the location

before the first node.
574
Exit How do you maintain this

while making progress?
Just take one step.
575
Exit
576
preCondInsert The location for the new
element has been found.
PostCondInsert The new element has been
inserted where it belongs.
postCondFind
577
preCondInsert The location for the new
element has been found.
PostCondInsert The new element has been
inserted where it belongs.
postCondInsert
Invariants  578
Bucket (Quick) Sort for Humans
Input: A pile of things to sort.
(Ex: Exams)
Output: Sort them
Requirements:
• Easy to execute by humans
• Only can have 7 piles on desk
• Can move one exam (or pile)
at a time.
• Fast – O(nlog(n)) time.
Likely the only algorithm
in this course which you will
execute by hand yourself. 579
Input: [A-Z]
Denotes an unsorted pile of exams

with last names starting in the range [A-Z]
580
[A-Z]
Psychology study:
Humans can only think about
5 things at once.
[A-E]
Is the first letter [F-K]
of the name in [L-O]
on the first exam [P-T]
[U-Z]
581
[A-Z]
[A-E] [F-K] [L-O] [P-T] [U-Z]
Bucket Sort
• Put exams one at a time
in the correct bucket. 582
[A-E]
[] [F-K]
[L-O]
sorted [P-T]
[U-Z]
A sorted pile (upside down)

• before all the rest
A stack of piles
• each pile unsorted
• all in one pile before all in next 583
[A-E]
[] [F-K]
[L-O]
sorted [P-T]
[U-Z]
[A] [B] [C] [D] [E]
Bucket Sort
• Put exams one at a time
in the correct bucket. 584
[A]
[] [B]
[C]
[D]
sorted [E]
[F-K]
[L-O]
[P-T]
[U-Z]
A stack of piles
[A]
[] [B]
[C]
[D]
sorted [E]
[F-K]
[L-O]
[P-T]
[U-Z]
[AA-AE][AF-AK][AL-AO] [AP-AT] [AU-AZ]
586
[AA-AE]
…
[]
[AU-AZ]
[B]
…
sorted
[E]
[F-K]
…
[U-Z]

A stack of piles
[AA-AE]
…
[]
[AU-AZ]
[B]
…
sorted
[E]
[F-K]
…
[U-Z]
[AA-AE]
588
[AF-AK]
…
[]
[AU-AZ]
[B]
…
sorted
[E]
[F-K]
…
[U-Z]
[AA-AE]
When sufficiently small
sort by hand sorted
589
[AF-AK]
…
[AA-AE]
[AU-AZ]
[B]
…
sorted
[E]
[F-K]
…
[U-Z]

A stack of piles
[AF-AK]
…
[AA-AE]
[AU-AZ]
[B]
…
sorted
[E]
[F-K]
…
[U-Z]
[AF-AK]
sorted
591
[AL-AO]
…
[AA-AK]
[AU-AZ]
[B]
…
sorted
[E]
[F-K]
…
[U-Z]

A stack of piles
[AL-AO]
…
[AA-AK]
[AU-AZ]
[B]
…
sorted
[E]
[F-K]
…
[U-Z]
[AL-AO]
[AP-AT]
[AU-AZ]
sorted
sorted
sorted
593
[B]
…
[AA-AZ]
[E]
[F-K]
sorted
…
[U-Z]
594
[B]
…
[A]
[E]
[F-K]
sorted
…
[U-Z]

A stack of piles
[B]
[A]
…
[E]
sorted [F-K]
…
[U-Z]
[BA-BE][BF-BK][BL-BO] [BP-BT] [BU-BZ]
596
[BA-BE]
…
[A]
[BU-BZ]
[C]
…
sorted
[E]
[F-K]
…
[U-Z]

A stack of piles
[A-ZT]
[ZU-ZZ]
sorted
[ZU-ZZ]
sorted
598
[A-Z]
sorted
Exit
599
Fast – O(nlog(n)) time.
• Each time you touch an exam,
the size of its pile goes
down by a factor of 5
• Hence I touch it only log5n times.
(Assuming names distributed
randomly through alphabet)
[A-Z]
[A-E] [F-K] [L-O] [P-T] [U-Z]
600
RadixSort 344
125
224
Input: A of stack of N punch cards. 125
225
Each card contains d digits. 333
325
Each digit between 0&(k-1) 134
Output: Sort the cards. 224 333
334 134
Bucket Sort Machine: 143 334
Select one digit 225 344
Separates cards into k piles. 325 143
wrt selected digit. 243 243
Stable sort: If two cards are the same for that digit,
their order does not change. 601
RadixSort
344 125 125
125 134 224
333 Sort wrt which 143 Sort wrt which 225
134 digit first? 224 digit Second? 325
224 225 134
334 The most 243 The next most 333
143 significant. 344 significant. 334
225 333 143
325 334 243
243 325 344
All meaning in first sort lost.
602
RadixSort
344 333 224
125 143 125
333 Sort wrt which 243 Sort wrt which 225
134 digit first? 344 digit Second? 325
224 134 333
334 The least 224 The next least 134
143 significant. 334 significant. 334
225 125 143
325 225 243
243 325 344
603
RadixSort
344 333 2 24
125 143 1 25
333 Sort wrt which 243 Sort wrt which 2 25
134 digit first? 344 digit Second? 3 25
224 134 3 33
334 The least 224 The next least 1 34
143 significant. 334 significant. 3 34
225 125 1 43
325 225 2 43
243 325 3 44
Is sorted wrt first i digits.
604
RadixSort
2 24 1 25
1 25 1 34
2 25 Is sorted wrt 1 43 Is sorted wrt
3 25 first i digits. 2 24 first i+1 digits.
3 33 2 25
1 34 2 43
3 34 These are in the
1 43 3 25 correct order
2 43 Sort wrt i+1st 3 33 because sorted
3 44 digit. 3 34 wrt high order digit.
i+1
3 44
605
RadixSort
2 24 1 25
1 25 1 34
2 25 Is sorted wrt 1 43 Is sorted wrt
3 25 first i digits. 2 24 first i+1 digits.
3 33 2 25
1 34 2 43
3 34 These are in the
1 43 3 25 correct order
2 43 Sort wrt i+1st 3 33 because was sorted &
3 44 digit. 3 34 stable sort left sorted.
i+1
3 44
606
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 3 3 3
Input: N records each labeled with a digit

between 0&(k-1).
Output: Stable sort the numbers.
Algorithm: Count to determine where records go.

Go through the records in order
putting them where they go.
607
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Stable sort: If two records are the same for that digit,
their order does not change.
Therefore, the 4th record in input with digit 1 must be
the 4th record in output with digit 1.
It belongs in output index 8, because
8 records go before it
ie 5 records with a smaller digit & Count
3 records with the same digit These608
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output:
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
# of records with digit v: 5 9 3 2
N records, k different values. Time to count? (N× k)

We have counted # of each value (N)
in the first i values. 609
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output:
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
# of records with digit < v: 0 5 14 17
N records, k different values. Time to count? (k2)
Too much
610
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output:
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
Computed
N records, k different values. Time to count? (k) 611
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
Location of first record
with digit v.
612
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 ? 1
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
Location of first record 0 5 14 17
with digit v.
Algorithm: Go through the records in order
613
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 1
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
Location of next record 0 5 14 17
with digit v.
614
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 1
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
615
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 1
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
616
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 1 1
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
617
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 1 1 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
618
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 1 1 1 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
619
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 1 1 1 1 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
620
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 1 1 1 1 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
621
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 1 1 1 1 1 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
622
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 1 1 1 1 1 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
623
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 1 1 1 1 1 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
624
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 1 1 1 1 1 1 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
625
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 1 1 1 1 1 1 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
626
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v: 0 1 2 3
with digit v.
Time = (N) (Counting Ops)
Total = (N+k) ×(log N + log k) (bit ops) 627
Radix/Counting Sort
Input: N numbers, each L bits long.
Output: Sort the numbers.
111101110100101000101
N 101001010100010110111
110100011010011110101
L
111 101 110 100 101 000 101
N 101 001 010 100 010 110 111
110 100 011 010 011 110 101
= log k (k will be set to N)
d = L / log k 628
Radix/Counting Sort
Input: N numbers, each L bits long.
Each card contains d digits.
Each digit between 0&(k-1)
Output: Sort the numbers.
Use Radix Sort: Sort wrt each digit using Counting Sort.
111 101 110 100 101 000 101

N 101 001 010 100 010 110 111 digit between 0&(k-1).
110 100 011 010 011 110 101
= log k
d = L / log k
629
Radix/Counting Sort
630
Radix/Counting Sort
Time = (Time of Radix Sort)

= (# of digits) × (Time of Counting Sort)
= L/log k × (N+k) × (log N + log k) bit ops
Set k to minimize time.

Wants k big. Really wants k small, but does not
care as long as k  N.
Set k=N
631
Radix/Counting Sort
Time = (Time of Radix Sort)

= (# of digits) × (Time of Counting Sort)
= L/log k × (N+k) × (log N + log k) bit ops
= L/log N × N × log N bit ops
= (L × N) bit ops = (n) bit ops
Size = N numbers, each L bits long.
= (L × N) = n
“Linear” Time
But sorting should take (N log N) time!!! 632
Radix/Counting Sort
Time = (L × N) bit ops  (N log N) bit ops
Size = N numbers, each L bits long.
= (L × N)
L => log N if you want N distinct numbers.
Merge or Quick Sort

Time = (N log N) comparisons = (n)
Size = Size of N numbers, each arbitrarily long.
= (N log N) bits.
633
Time Complexity of Sorting
Merge, Quick, and Heap Sort can sort N numbers

using O(N log N) comparisons between the values.
Theorem: No algorithm can sort faster.
634
A Upper Bound on Sorting
MergeSort Sorting 10 n log(n)
$ A, " I, [ A(I) = P(I) and Time(A,I) ≤ Tupper(|I|)]
I have an algorithm A that I
claim works and is fast.
Oh yeah, I have an input I for
which it does not .
I win if A on input I gives
• the right output and
• runs fast enough.
I win!
635
A Lower Bound on Sorting
??? Sorting 0.1 n log(n)
" A, $ I, [ A(I) = P(I) or Time(A,I) ³ Tlower(|I|)]
I have an algorithm A that I
claim works and is fast.
Oh yeah, I have an input I for
which it does not .
I win if A on input I gives
• the wrong output or
• runs slow.
I win!
636
Narrowing the Search Space Loop Invariant
• If the key is contained in the original list, then
it is contained in the narrowed subspace.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95

left half. right half.
637
instance on which
the algorithm fails
I am looking for an instance I
on which the algorithm A fails.
I give you an algorithm A

that I claim sorts faster.
638
instance on which
the algorithm fails
I maintain a set of instances

Oh dear!
The first t time steps of my
1 2 3 4 5 6 7 8
algorithm A are the same
3 2 5 8 6 1 4 7
given any of these
4 7 5 2 8 1 3 6 instances!!!
5 4 2 3 6 8 1 7
639
My t+1st time step
79 km 75 km • a7 < a2
• 5th bit of a7 + a2
• Any yes/no question
I Imaintain
keep thealarger
set ofhalf.
instances
Oh dear!
1 2 3 4 5 6 7 8
algorithm A are the same
3 2 5 8 6 1 4 7
given any of these
4 7 5 2 8 1 3 6 instances!!!
5 4 2 3 6 8 1 7
Now, its t+1st step is the same.
640
Initially, I have n! Permutations.
After t time steps I have N!/2t
/2 factors each at least N/2. N
N! = 1 × 2 × 3 × … × N/2 × … × N
N factors each at most N.
N!  NN
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
641
We exit when there are two instances.

Exit T = log(N!) = N log(N)
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
642
Exit
I give you algorithm A
I must output an instance I claim it sorts.
4 7 5 2 8 1 3 6
on which alg A either

• Takes too much time
• Or gives the wrong answer
643
Exit
I must output an instance
4 7 5 2 8 1 3 6

Oops!
Case 1: The algorithm does not stop

at time T on these two instances.
Exit T = log(N!) = Q(N log(N)).
644
Exit
Oops!
4 7 5 2 8 1 3 6 I must give
5 4 2 3 6 8 1 7 the wrong
answer on
• Takes too much time one of these!
Case 2: The algorithm stops at time T

and gives an answer.
We exit when there are two instances
Exit
and these need different answers.
The first T time steps of alg A are the
same on these two instances
and hence the same answer is given. 645
Theorem: For every sorting algorithm A,

on the worst case input instance I,
Q(N log2 N) comparisons (or other bit operations)
need to be executed to sort N objects.
" A, $ I, A(I) = P(I) or Time(A,I) ³ N log2 N

Proof: Prover/Adversary Game
646
End Iterative Algorithms
Math Review
Recursive Algorithms
647


648
• Sorting
Values:
– Input: Indexes:
23 67 34 21 87 3 13 11
1 2 3 4 5 6 7 8
Values: 3 11 13 21 23 34 67 87
– Output: Indexes: 6 8 7 4 1 3 2 5
• Lower Bound:
– Input: An Algorithm for Sorting A
Values: 23 67 34 21 87 3 13 11
– Output: An instance I Indexes: 1 2 3 4 5 6 7 8
Values: 11 3 13 21 23 34 67 87
Indexes: 8 6 7 4 1 3 2 5
649
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
650
I claim it sorts.
Values: 23 67 34 21 87 3 13 11
Indexes: 1 2 3 4 5 6 7 8

It might as well be a
permutation of 1..N
(indexes not shown)
4 7 5 2 8 1 3 6
651
I claim it sorts.
Need to know
what the algorithm does
before we can know
what input to give it.
Break this cycle,

one iteration at a time.
Need to give the

algorithm an input
before we can know
what the algorithm does with the input. 652
Restrict the search space
653
I maintain a set of instances

(permutations of 1..n)
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Oh dear!
algorithm A are the same given
any of these instances!!!
654
Initially, I consider all N!

permutations of 1..N
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Oh dear!
The first 0 time steps of my
655
Initially, I consider all N!

permutations of 1..N
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
The measure of progress

79 km km is the number of instances.
Initially, there are N! of them.
656
My t+1 time step st
• a7 < a2
• Any yes/no
question
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Because the first t time steps

are the same, what the alg knows
is the same.
Hence, its t+1st step is the same.
657
My t+1 time step st
• a7 < a2
I partition my instances based on
• Any yes/no
what they do on this t+1st step.
question
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
Because the first t time steps

are the same, what the alg knows
is the same.
Hence, its t+1st step is the same.
658
I keep the larger half.
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
4 7 5 2 8 1 3 6
5 4 2 3 6 8 1 7
659
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7
Oh dear!
The first t+1 time steps of my
660
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7

79 km After t time steps I have N!/2t
661
1 2 3 4 5 6 7 8
3 2 5 8 6 1 4 7

Exit We exit when there are two instances.

T = log(N!) 662
/2 factors each at least N/2.

N
N! = 1 × 2 × 3 × … × N/2 × … × N
N factors each at most N.
N!  NN
Exit We exit when there are two instances.

T = log(N!) = Q(N log(N)). 663
Exit
I must output an instance I claim it sorts.
Values: 23 67 34 21 87 3 13 11
Indexes: 1 2 3 4 5 6 7 8

664
Exit
Values: 23 67 34 21 87 3 13 11
Indexes: 1 2 3 4 5 6 7 8

Oops!
Case 1: The algorithm does not stop

at time T on these two instances.
Exit T = log(N!) = Q(N log(N)).
665
Exit
Oops!
I must give
Values: 23 67 34 21 87 3 13 11
Indexes: 1 2 3 4 5 6 7 8
the wrong
answer on
• Takes too much time one of these!
Case 2: The algorithm stops at time T

and gives an answer.
We exit when there are two instances
Exit
and these need different answers.
The first T time steps of alg A are the
same on these two instances
and hence the same answer is given. 666


667
Find Errors
• PreCond: i=0 • Loop Invariant:
• PostCond: i=131 i is increasing
• Step: i = i+2
• Exit Cond: i=131
LI is an action not a picture.
668
Find Errors
• PostCond: i=131 ??
• Step: i = i+2
<preCond>
<loop-invariant>
codeA

LI has no real content.
669
Find Errors
• Step: i = i+2
Exit
<loop-invariantt>
codeB

LI has no real content.
670
Find Errors
• Step: i = i+2
Clean up loose ends
<loop-invariant>
Exit
codeC

<exit Cond> = <postCond>
NEVER DO THIS! 671
Find Errors
• Step: i = i+2
Exit
79 km 79 km 75 km km
Trivially true
Let measure be the value of i
(or 131-i)
672
Find Errors
• Step: i = i+2
? 0 km Exit
Sufficient progress is made
but alg fails to exit!
673
Find Errors
• Step: i = i+2
≥
? 0 km Exit This change ensures we exit.
674
Find Errors
• PostCond: i=131 ?? i is odd
• Step: i = i+2
• Exit Cond: i≥131
675
Find Errors
• PostCond: i=131 i is odd
• Step: i = i+2
<preCond> i=0
codeA ? i = i+1
i is odd
<loop-invariant>
676
Find Errors
• PostCond: i=131 i is odd & i ≤ 132
• Step: i = i+2
Clean up loose ends
<loop-invariant> i is odd & i ≤ 132
Exit
<exit Cond> i≥131
codeC
i=131
<postCond> 677
Find Errors
• PostCond: i=131 i is odd & i ≤ 132
• Step: i = i+2
<loop-invariant> i’ is odd and ≤132
Exit
¬<exit Cond> i’ < 131
codeB i’’ = i’+2
i’’ is odd and ≤132
<loop-invariant> 678
Invariants Progress
79 km

Exit
Exit

Exit
79 km 75 km
679
More Pointers
680
More Pointers
2039 0
element next
head 2039
8715 9321
element next
temp 8715
2039 8715
element next
Name Location Value x y 9321
of box of box in box

x x=y
Location Value
y
of box x in box y
red = stepped on put green = go
681
More Pointers
2039 0
element next
head 2039
8715 9321
element next
temp 8715
element next
9321
Name Location Value
of box of box in box
head head.next = temp.next
Location Value
head.next of box x in box y
temp.next
red = stepped on put green = go
682
More Pointers
Which values change?

What is their new value?
Who has that value?
What is the code?
done = todo;
The right hand side of the “=”
specifies a memory location.
So does its left hand side.
The action is to put the value 1
1 1 contained in the first
1 into the second. 1 683
More Pointers

Who has that value?
What is the code?
todo = todo.next;
A classic line
2 to move along 2
2 a linked list. 2
2 684
More Pointers

Who has that value?
What is the code?
todo.next = done;
3 3
3 3
3
685
More Pointers
done = todo;
todo = todo.next;
todo.next = done;
Oops!
We don’t want to
drop the old values
getting the new.
23
13
12 686
More Pointers
No.
done = todo;
The value of
todo changes. todo = todo.next;
todo.next = done;
done-old = done;
todo-old = todo;
todo-next-old = todo.next;
done = todo-old;
23
13 todo = todo-next-old;
12 todo.next = done.old; 687
More Pointers
No.
done = todo;
The value of
todo changes. todo = todo.next;
todo.next = done;
done-old = done;
todo-old = todo;
done = todo-old;
23
13 todo = todo-next-old;
12 todo-old.next = done-old; 688
More Pointers
Good
done = todo;
But fewer
todo = todo.next;
lines?
todo.next = done;
done-old = done;
todo-old = todo;
todo.next = done-old;
23
13 done = todo-old;
12 todo = todo-next-old; 689
More Pointers
Good
done = todo;
But fewer
todo = todo.next;
lines?
todo.next = done;
Node temp = todo.next;
todo.next = done;
Can now be
stepped on.
10
0 1
690
More Pointers
Good
done = todo;
But fewer
todo = todo.next;
lines?
todo.next = done;
todo.next = done;
done = todo;
Can now be
stepped on.
12
2 691
More Pointers
Good
done = todo;
But fewer
todo = todo.next;
lines?
todo.next = done;
todo.next = done;
done = todo;
todo = temp;
Can now be
stepped on.
3
23 692
Cake Cutting
Precondition: A cake and the preferences of n people.
I like the red stuff on the right the most and

that cone in the middle section second.
Area ≥ 1/n
I hate that middle thing.

Area ≥ 1/n
The area under a piece of cake states how much the person values it.
Area under the entire cake is 1.
Postcondition: Partition the cake among the n people
so that each gets an amount that SHE personally values ≥ 1/n.
693
Cake Cutting
Area ≥ i/n
Area ≥ i/n
loop Some n-i people have pieces that they are happy with.
inv: The remaining i people value the remaining cake ≥ i/ . 694
Cake Cutting
Area = 1/n i-1

Area ≥ /n
i
i-1
Area ≥ /ni
Area = /n
1
Each of the remaining i people,

says how much of the remaining cake they need to get = 1/n.
We givei-1it to the guy who wants the least.
loop Some n-i people have pieces that they are happy with.
i-1
i-1
inv: The remaining i people value the remaining cake ≥ i/ . 695
Matrix Search
Precondition: nn matrix, each row and each column sorted. A key=5.
PostConditions: Find key in list (if there).
0 2 2 4 6 Search row in log(n) time
4 8 9 Search n rows in n log(n) time
Can you do it faster?
Can you do it in O(log(n)) time?
6
8 9
696
Matrix Search
0 Read the middle value.
6 If too big?
4 Now what?
6
6
4
697
Matrix Search
0 Maintain a sub-list.
6 If the key is contained in the original list,
4 then the key is contained in the sub-list.
6
Now what?
6
4 What is the running time
till all possible 5’s are eliminated?
9
What would be a prettier
narrowed search space?
698
Matrix Search
If the key is contained in the original list,
6 Look at the bottom right corner.
Does not feel right.
699
Matrix Search
4 then the key is contained in the sub-list.
Look at the top right corner.
700
Matrix Search
6 Look at the top right corner.
This maintains the loop invariant.
701
Matrix Search
Running time?
9 - Each iteration eliminates a row or col.
702
Matrix Search
Running time?
703
Matrix Search
Running time?
704
Matrix Search
4 This maintains the loop invariant.
Running time?
705
Matrix Search
6
Running time?
706
Matrix Search
6
Running time?
707
Matrix Search
4
Running time?
708
Matrix Search
4 Running time?
709
Matrix Search
Running time?
4 9 - Each iteration eliminates a row or col.
710
Matrix Search
Running time?
- The # rows + # col = n+n
- Time = 2n-1
711
Matrix Search
0 4 6 Note that this is property sorted.
5
4 6 Even if a 4 or 6 is changed to a 5.
4 6
4 6 What happens if when searching
4 6
5
you do not check each 4&6?
4 6
4 6 How much time is needed?
6 9 2n-1.
712

01 Iterative

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 Iterative

Uploaded by

Copyright:

Available Formats

Iterative Algorithms

& Loop Invariants

“The Rules taught should be

No problem. I know how to multiply.

Is the black the form and the green the background?

I can get lost!!! 7

If the computation does not meet the assertion

Defn of Correctness: <assertion0>

Defn of Correctness: <assertion0>

Defn of Correctness: <assertion0>

Defn of Correctness: <assertion0>

Defn of Correctness: <assertion0>

Each path through this code

black and the beginning of the next.

You need to define some

• A good way to structure many

I implored you to not worry

I implored you to not worry

I implored you to not worry

I can get lost!!! 53

I can get lost!!! 54

m is max in {A[1]..A[i]}” Max of 4 is 4

m is max in {A[1]..A[i]}” Max of  is ?

m is max in {A[1]..A[i]}” Max of 4 is 4

m is max in {A[1]..A[i]}” Max of 4 is 4

Exit Obtaining Postcondition:

S(0) S(1) S(2) S(3) S(4) S(2) S(3) S(4) …

My goal is to prove "i

Establishing the loop invariant:

To prove this using the game,

I must prove S(i).

Why can I then conclude "i S(i)???

My goal is to prove "i

S(0) S(1) S(2) S(3) S(4)

S(0) S(1) S(2) S(3) S(4)

Being an assumption, I can assure you that the

But he does not know anything else.

My goal is to prove that the code before the loop

Being an assumption, I can assure you that we are at

This tells me that we have two variables m and i and

But he does not know anything else.

I can assure you that the loop invariant is true,

I can assure you that the loop invariant is true,

exit Cond: i=n. code: Return m.

My goal is to prove that in the end, the post condition is true,

I will try to translate.

Exit Maintaining Loop Invariant

Define Step Define Exit Condition Maintain Loop Inv

Initial Conditions Ending

exit when (car at end) Actions?

Define Step Define Exit Condition Maintain Loop Inv

Initial Conditions Ending

Let x(t) denote its distance from rest.

Let x(t) denote its distance from rest. Newton: F=MA

Dude! You can’t do induction on the reals!

My goal is to plot x(t) for t≥0.

S(0) simply states that we know

My goal now is to prove

I help you by assuming S(T).

Then I prove S(T+ε). Maintainin

I help you by assuming S(T).

•x(T+ε) = x(T) + velocity(T)  ε

It is so important to balance units.