Basic Techniques for Design and Analysis of Algorithms

EDWARD M. REINGOLD
Department of Computer Science, University of Illinois at Urbana-Champaign ^reingold@cs.uiuc.edu&

ANALYZING ALGORITHMS 10. }

11.
It is convenient to classify algorithms based
12. void insertionSort (int n,
on the relative amount of time they require: float x[ ]) {
how fast does the time required grow as the
13. // Sort x[1] . . . x[n]
size of the problem increases? For example,
in the case of arrays, the “size of the prob- 14. if (n . 1) {
lem” is ordinarily the number of elements 15. insertionSort (n21, x);
in the array. If the size of the problem is 16. insert (x, n, x[n]);
measured by a variable n, we can express 17. }
the time required as function of n, T(n). 18. }
When this function T(n) grows rapidly, the
algorithm becomes unusable for large n; To determine the time required in the
conversely, when T(n) grows slowly, the worst case to sort n elements with inser-
algorithm remains useful even when n be- tionSort, we let tn be the time to sort n
elements and derive and solve a recur-
comes large. The time required by an algo-
rence relation for tn. We have
rithm is often analyzed by finding and solv-
ing a recurrence relation that describes the
time required by the algorithm.

AN EXAMPLE: INSERTION SORT

tn H Q ~1!
t n21 1 s n21 1 Q ~ 1 !
if n 5 1 ,
otherwise ,
where sm is the time required to insert
How can we reorder the elements of an an element in place among m elements
array of n values x[1], x[2], . . ., x[n] so using insert. The value of sm is also
that they are in perfect order so that x[1] given by a recurrence relation:

H
# x[2] # . . . # x[n]? The simplest way to
put the values in order is to mimic what Q ~1! if m 5 1 ,
we might do by hand: take item after sm
item and insert each one into the proper s m21 1 Q ~ 1 ! otherwise .
place among those items already inserted: With a bit of algebra we discover that
1. void insert (float x[ ], int i, tn 5 Q(n2). The analysis of the average
float a) { behavior is nearly identical; only the con-
2. // Insert a into x[1] . . . stants hidden in the Q-notation change.
x[i]
3. // x[1] . . . x[i21] are sort- DIVIDE AND CONQUER
ed; x[i] is unoccupied
We can design better sorting methods
4. if (i 55 1 i x[i21] d a)
using the idea of divide-and-conquer, in
5. x[i] 5 a; which we decompose a problem into
6. else { subproblems that resemble the original
7. x[i] 5 x[i21]; problem on a reduced scale. For exam-
8. insert (x, i21, a); ple, we can sort by splitting the n values
9. } arbitrarily into two piles of n/2 values

Copyright © 1996, CRC Press.

This work was supported in part by the National Science Foundation, grant number CCR-93-20577.

ACM Computing Surveys, Vol. 28, No. 1, March 1996

20 • Edward M. Reingold
each, sort each of the piles separately,
and then merge the two piles into a
single sorted pile. This sorting tech-
niques is called merge sort. Let T(n) be
the time required by merge sort for sort-
ing n values. The time needed to do the
merging is proportional to the number
of elements being merged, so that
T ~ n ! 5 cn 1 2T ~ n/ 2 ! ,
because we must sort the two halves
(time T(n/2) each) and then merge (time
proportional to n). The growth rate of
T(n) is Q(n log n).
LOWER BOUNDS
Can we find sorting algorithms that take
time less than Q(n log n)? The answer is
no if we are restricted to sorting algo-
rithms that derive their information from
comparisons between the values of ele-
ments. The flow of control in such sorting
algorithms can be viewed as binary trees
in which there are n! leaves, one for every
possible sorted output arrangement. Be-
cause a binary tree with height h can
have at most 2h leaves, it follows that the
height of a tree with n! leaves must be at
least log2 n! 5 Q(n log n). Because the
height of this tree corresponds to the
longest sequence of element comparisons
possible in the flow of control, any such
sorting algorithm must, in its worst case,
use time proportional to n log n.
DYNAMIC PROGRAMMING
In designing algorithms to solve optimiza-
tion problems, we need to make the opti-
mal (lowest cost, highest value, shortest
distance, and so on) choices from among a
large number of alternative solutions; dy-
namic programming is an organized way
to find an optimal solution by systemati-
cally exploring all possibilities without
unnecessary repetition. Often dynamic
programming leads to efficient, polynomi-
al-time algorithms for problems that ap-
pear to require searching through expo-
nentially many possibilities.
Like the divide-and-conquer method,
dynamic programming is based on the
ACM Computing Surveys, Vol. 28, No. 1, March 1996
observation that many optimization prob-
lems can be solved by solving similar sub-
problems and then composing the solu-
tions of those subproblems into a solution
for the original problem. In addition, the
problem is viewed as a sequence of deci-
sions, each decision leading to different
subproblems; if a wrong decision is made,
a suboptimal solution results, so it is nec-
essary to account for all possible deci-
sions. To eliminate duplicate computa-
tions that occur in such sequences of
decisions, we use the idea of caching the
solutions to subproblems.
GREEDY HEURISTICS
Optimization problems always have an
objective function to be minimized or
maximized, but it is not often clear
what steps to take to reach the optimum
value. We may use dynamic program-
ming to examine systematically all pos-
sible choices, but perhaps there is a
simple rule that leads directly to good
choices at each step. Such an approach
may be less time-consuming than dy-
namic programming, but decisions are
made that are locally optimum, though
perhaps not globally optimum. But such
a “greedy” sequence of locally optimum
choices can lead to a globally optimum
solution in some circumstances.
How do we know that such a process
leads to an optimal solution? The key is
to assume, by way of contradiction, that
the greedy solution found is not opti-
mum. In this case, the greedy strategy
must have erred in one of its choices, so
we look at the first error this strategy
made. We then show that this “first
error” is not really an error at all, so
that we are forced to conclude that the
greedy strategy never errs.
We can apply the greedy heuristic to
many optimization problems, and even if
the results are not optimal, they are often
quite good. For example, in the n-city
traveling salesman problem, we can get
near-optimal tours in time O(n2) when
the intercity costs are symmetric and sat-
isfy the triangle inequality. The closest
insertion algorithm starts with a “tour”
consisting of a single, arbitrarily chosen
 Basic Techniques for Design and Analysis of Algorithms
city, and successively inserts the remain-
ing cities to the tour, making a greedy
choice about which city to insert next and
where to insert it: the city chosen for
insertion is the city not on the tour but
closest to a city on the tour; the chosen
city is inserted adjacent to the city on the
tour to which it is closest.
ACKNOWLEDGMENTS
The comments of Ken Urban are gratefully ac-
knowledged.
REFERENCES
CORMEN, T. H., LEISERSON, C. E., AND RIVEST, R.
L. 1990. Introduction to Algorithms.
McGraw-Hill, New York.
GREENE, D. H. AND KNUTH, D. E. 1990.
• 21
Mathematics for the Analysis of Algorithms,
3rd ed. Birkhäuser, Boston.
KNUTH, D. E. 1973. The Art of Computer Pro-
gramming, Vol. 1: Fundamental Algorithms,
2nd ed. Addison-Wesley, Reading, MA.
KNUTH, D. E. 1973. The Art of Computer Pro-
gramming, Vol. 3: Sorting and Searching, Ad-
dison-Wesley, Reading, MA.
LEUKER, G. S. 1980. Some techniques for solving
recurrences, ACM Comput. Surv. 12, 419 – 436.
REINGOLD, E. M. AND HANSEN, W. J. 1986. Data
Structures in Pascal. Little, Brown, Boston.
REINGOLD, E. M., NIEVERGELT, J., AND DEO,
N. 1977. Combinatorial Algorithms: The-
ory and Practice. Prentice-Hall, Englewood
Cliffs, NJ.
ROSENCRANTZ, D. J., STEARNS, R. E., AND LEWIS, P.
M. 1977. An analysis of several heuristics
for the traveling salesman problem. SIAM J.
Comput. 6, 563–581.
ACM Computing Surveys, Vol. 28, No. 1, March 1996