Unit 2

Advanced Algorithmic Analysis

Content
Amortized analysis.
Online and offline algorithms.
Randomized algorithms.
Dynamic programming.
combinatorial optimization.
Amortization
1. The gradual elimination of liability, such as a mortgage, in
regular payments over a specified period of time. Such
payments must be sufficient to cover both prinicpal and
interest.

2. Writing off an intangible assets investment over the
projected title of the assets.
Amortized analysis An
intro
It is a strategy for analyzing a sequence of operations to
show that the average cost per operation is small, even
though a single operation with the sequence might be
expensive.
To find
Average running time per operation = f ( worst case sequence of operations).
Not an algorithmic technique.
More accurate technique for analyzing algorithm under some
conditions.
It guarantees the time per operation over worst case
performance.
The basic idea is that a worst case operation can alter the
state in such a way that worst case can not occur again for a
long time, thus amortizing its cost.
Introduction
In average case analysis: averaging over all possible inputs.
In probabilistic analysis: averaging over all possible random
choices.
In amortized analysis: averaging over a sequence of
operations.
Amortized analysis assumes worst case input and not
typically do not allow random choices.
Amortized Analysis
Aggregate Analysis
Accounting Method /
Taxation Method
Potential Method
Introduction
Aggregate Analysis:
determines the upper bound T(n) on the total cost of sequence of
n operations,
then calculate the average cost to be T (n)/n.
Accounting Method / Taxation Method:
Assign each type of operation an (different) amortized cost.
overcharge some operations.
store the overcharge as credit on specific objects.
then use the credit for compensation for some later operations.
Potential Method:
Same as accounting method
But store the credit as potential energy and as a whole.
Aggregate Analysis
Show that for all n, a sequence of n operations take worst-case
time T(n) in total.

In the worst case, the average cost, or amortized cost , per
operation is T(n)/n.

The amortized cost applies to each operation, even when there
are several types of operations in the sequence.
Example: Two-Stack System
Suppose there are two stacks called A and B, manipulated by
the following operations:
push(S, d): Push a datum d onto stack S.
Real Cost = 1.
multi-pop(S, k): Removes min(k, |S|) elements from stack
S.
Real Cost = min(k, |S|).
transfer(k): Repeatedly pop elements from stack A and
push them onto stack B. until either k elements have been
moved, or A is empty.
Real Cost = # of elements moved = min(k, |A|).

Illustration
A
B
s n push operations
s n elements
transferred
s n elements popped
Aggregate Method
For a sequence of n operations, there are on data pushed
into either A or B. Therefore there are on data popped out
from either A or B and on data transferred from A to B. Thus,
the total cost of the n operations is o3n. Thus.
Operation Real Cost Amortized Cost
Push(A, d) 1 3
Push(B, d) 1 3
Multipop(A, k) min(k, |A|) 3
Multipop(B, k) min(k, |B|) 3
Transfer(k) min(k, |A|) 3
Example for amortized analysis
Stack operations:
PUSH(S,x), O(1)
POP(S), O(1)
MULTIPOP(S,k), min(s,k)
while not STACK-EMPTY(S) and k>0
do POP(S)
k=k-1
Let us consider a sequence of n PUSH, POP, MULTIPOP.
The worst case cost for MULTIPOP in the sequence is O(n),
since the stack size is at most n.
thus the cost of the sequence is O(n
2
). Correct, but not tight.
Example for amortized
analysis
Sequence of n push, pop, Multipop operations
Worst-case cost of Multipop is O(n)
Have n operations
Therefore, worst-case cost of sequence is O(n
2
)

Observations
Each object can be popped only once per time that its
pushed
Have <= n pushes => <= n pops, including those in Multipop
Therefore total cost = O(n)
Average over n operations => O(1) per operation on average

Notice that no probability involved
Aggregate Analysis
In fact, a sequence of n operations on an initially empty stack
cost at most O(n). Why?
Each object can be POPed only once (including in MULTIPOP) for each
time it is PUSHed.
#POPs is at most #PUSHs, which is at most n.
Thus the average cost of an operation is O(n)/n = O(1).
Amortized cost in aggregate analysis is defined to be average
cost.
Incrementing a Binary Counter
k-bit Binary Counter: A[0..k1]
INCREMENT(A)
1. i 0
2. while i < length[A] and A[i] = 1
3. do A[i] 0 reset a bit
4. i i + 1
5. if i < length[A]
6. then A[i] 1 set a bit

=

=
1
0
2 ] [
k
i
i
i A x
k-bit Binary Counter
Ctr A[4] A[3] A[2] A[1] A[0]
0 0 0 0 0 0
1 0 0 0 0 1
2 0 0 0 1 0
3 0 0 0 1 1
4 0 0 1 0 0
5 0 0 1 0 1
6 0 0 1 1 0
7 0 0 1 1 1
8 0 1 0 0 0
9 0 1 0 0 1
10 0 1 0 1 0
11 0 1 0 1 1
k-bit Binary Counter
Ctr A[4] A[3] A[2] A[1] A[0] Cost
0 0 0 0 0 0 0
1 0 0 0 0 1 1
2 0 0 0 1 0 3
3 0 0 0 1 1 4
4 0 0 1 0 0 7
5 0 0 1 0 1 8
6 0 0 1 1 0 10
7 0 0 1 1 1 11
8 0 1 0 0 0 15
9 0 1 0 0 1 16
10 0 1 0 1 0 18
11 0 1 0 1 1 19
Worst-case analysis
Consider a sequence of n increments. The worst-
case time to execute one increment is O(k).
Therefore, the worst-case time for n increments is
n O(k) = O(n k).
WRONG! In fact, the worst-case cost for
n increments is only O(n) O(n k).
Lets see why.
Note: Youd be correct
if youd said O(n k).
But, its an overestimate.
Tighter analysis
Ctr A[4] A[3] A[2] A[1] A[0] Cost
0 0 0 0 0 0 0
1 0 0 0 0 1 1
2 0 0 0 1 0 3
3 0 0 0 1 1 4
4 0 0 1 0 0 7
5 0 0 1 0 1 8
6 0 0 1 1 0 10
7 0 0 1 1 1 11
8 0 1 0 0 0 15
9 0 1 0 0 1 16
10 0 1 0 1 0 18
11 0 1 0 1 1 19
A[0] flipped every op n
A[1] flipped every 2 ops n/2
A[2] flipped every 4 ops n/2
2

A[3] flipped every 8 ops n/2
3


A[i] flipped every 2
i
ops n/2
i

Total cost of n operations
Tighter analysis (continued)
) (
2
0

2
1


lg
0
2

n
n
i
i
n
n
i
i
n
O =
=

=
<

=
(

=
(

Cost of n increments
.
Thus, the average cost of each increment operation
is O(n)/n = O(1).
Vellore to Tirupathi Trip
Travel charge for Vellore to Tirupathi: Rs 100/-
Travel charge for Tirupathi to Vellore : Rs 100/-
Total travel charge: Rs 200 /-
Vellore to Tirupathi (Round
Trip)
Travel charge Rs 200/-
Travel charge for Tirupathi to Vellore : Rs 0/-
Total travel charge: Rs 200 /-
Accounting / Taxation Method:
Assign different charges to different operations.
Charges may be more or less than actual cost.
CHARGING amount of operation amortized cost.
If (amortized cost > actual cost)
Difference credits, saved in specific objects.
If (amortized cost < actual cost)
Credits is used.
Total credit = -ve.
Sum of amortized cost is upper bound for the actual total cost
of the operations.
Let




- amortized cost of i
th
operation

=1

=1

Accounting method
Charge i
th
operation a fictitious amortized cost
i
, where $1 pays
for 1 unit of work (i.e., time).
Assign different charges (amortized cost ) to different
operations
Some are charged more than actual cost
Some are charged less
This fee is consumed to perform the operation.
Any amount not immediately consumed is stored in the bank for
use by subsequent operations.
The bank balance (the credit) must not go negative!
We must ensure that

= =
s
n
i
i
n
i
i
c c
1 1

for all n.
Thus, the total amortized costs provide an upper bound on the
total true costs.
A Simple Example:
Accounting
3 ops:
Push(S,x) Pop(S) Multi-pop(S,k)
Assigned
cost:
2 0 0
Actual cost: 1 1 min(|S|,k)
Push(S,x) pays for possible later pop of x.
1
1
1
1
1
1
1
1
1
0
1
1
0
0
0
Stack Example: Accounting
Methods
When pushing an object, pay $2
$1 pays for the push
$1 is prepayment for it being popped by either pop or
Multipop.
Since each object has $1, which is credit, the credit can
never go negative
Therefore, total amortized cost = O(n), is an upper bound
on total actual cost
Example:
Accounting analysis of
INCREMENT
Charge an amortized cost of $2 every time a bit is set from 0 to
1
$1 pays for the actual bit setting.
$1 is stored for later re-setting (from 1 to 0).
At any point, every 1 bit in the counter has $1 on it that pays for
resetting it. (reset is free)
0 0 0 1
$1
0 1
$1
0
0 0 0 1
$1
0 1
$1
1
$1

0 0 0 1
$1
1
$1
0 0
Cost = $2
Cost = $2
Incrementing a Binary Counter
When Incrementing,
Amortized cost for line 3 = $0
Amortized cost for line 6 = $2
Amortized cost for INCREMENT(A) = $2
Amortized cost for n INCREMENT(A) = $2n =O(n)
INCREMENT(A)
1. i 0
2. while i < length[A] and A[i] = 1
3. do A[i] 0 reset a bit
4. i i + 1
5. if i < length[A]
6. then A[i] 1 set a bit
Accounting analysis
(continued)
Key invariant: Bank balance never drops below 0. Thus, the
sum of the amortized costs provides an upper bound on the sum
of the true costs.
i 1 2 3 4 5 6 7 8 9 10
#1s

1 1 2 1 2 2 3 1 2 2
c
i


1 2 1 3 1 2 1 4 1 2

i
2 2 2 2 2 2 2 2 2 2
bank
i
1 1 2 1 2 2 3 1 2 2
Potential method
IDEA: View the bank account as the potential energy of the
dynamic set.
Framework:
Start with an initial data structure D
0
.
Operation i transforms D
i1
to D
i
.
The cost of operation i is c
i
.
Define a potential function u : {D
i
} R,
such that u(D
0
) = 0 and u(D
i
) > 0 for all i.
The amortized cost
i
with respect to u is defined to be

i
= c
i
+ u(D
i
) u(D
i1
).
Potential method II

Like the accounting method, but think of the credit as
potential stored with the entire data structure.
Accounting method stores credit with specific objects
while potential method stores potential in the data
structure as a whole.
Can release potential to pay for future operations
Most flexible of the amortized analysis methods.
Understanding potentials

i
= c
i
+ u(D
i
) u(D
i1
)
potential difference Au
i
If Au
i
> 0, then
i
> c
i
. Operation i stores work
in the data structure for later use.
If Au
i
< 0, then
i
< c
i
. The data structure
delivers up stored work to help pay for operation
i.
Amortized costs bound the true
costs
The total amortized cost of n operations is
( )

=

=
u u + =
n
i
i i i
n
i
i
D D c c
1
1
1
) ( ) (
Summing both sides.
The total amortized cost of n operations is
( )
) ( ) (
) ( ) (
0
1
1
1
1
D D c
D D c c
n
n
i
i
n
i
i i i
n
i
i
u u + =
u u + =

=
=

=
The series telescopes.
Amortized costs bound the true costs
The total amortized cost of n operations is
( )

=
=
=

=
>
u u + =
u u + =
n
i
i
n
n
i
i
n
i
i i i
n
i
i
c
D D c
D D c c
1
0
1
1
1
1
) ( ) (
) ( ) (
since u(D
n
) > 0 and
u(D
0
) = 0.
Amortized costs bound the true costs
Stack Example: Potential
Define: |(D
i
) = #items in stack Thus,
|(D
0
)=0.


Plug in for operations:
Push:
i
= c
i
+ |(D
i
) - |(D
i-1
)
= 1 + j - (j-1)
= 2
Pop:
i
= c
i
+ |(D
i
) - |(D
i-1
)
= 1 + (j-1) - j
= 0
Multi-pop:
i
= c
i
+ |(D
i
) - |(D
i-1
)
= k + (j-k) - j k=min(|S|,k)
= 0
Thus O(1) amortized
time per op.
Potential analysis of
INCREMENT
Define the potential of the counter after the i
th
operation
by u(D
i
) = b
i
, the number of 1s in the counter after the
i
th
operation.
Note:
u(D
0
) = 0,
u(D
i
) > 0 for all i.
Example:
u = 2
0 0 0 1 0 1

0
0 0 0 1
$1
0 1
$1
0 Accounting method)
(
Calculation of amortized costs
The amortized cost of the i
th
INCREMENT is

i
= c
i
+ u(D
i
) u(D
i1
)
= (t
i
+ 1) + (1 t
i
)
= 2
Assume i
th
INCREMENT resets t
i
bits (in line 3).
Actual cost c
i
= (t
i
+ 1)
Number of 1s after i
th
operation: b
i
= b
i

1
t
i
+ 1
Therefore, n INCREMENTs cost O(n) in the worst case.
Topics to be Discussed Next
Randomized algorithms.