ME subject

© All Rights Reserved

14 views

ME subject

© All Rights Reserved

- Exercises of Design & Analysis
- Optimal-Storage-on-Tapes
- MQL4 Programming
- november2002paper2
- (2) How Should I Practice So That I Will Be at a Level Where I Can Approach TopCoder's Div1-500 Problems With Confidence
- (6) What is the Maths Require for the Competitive Programming_ - Quora
- 07 Good Programming Style
- Implementing Quicksort Programs
- Tmp 3078
- Scimakelatex.95793.Md.+Sharafat+Jamil
- GATE Syllabus
- pajek
- Designing Good Algorithms for MapReduce and Beyond
- Scimakelatex.15413.Anon.mous
- Lecture 3
- Meidan Filter Doc
- Es 15 Libro
- Electronic Homework
- Analyzing B-Trees and Online Algorithms
- The Effect of Autonomous Information on Electrical Engineering

You are on page 1of 36

ANALYSIS O

OF

ALGORITHMS

HM

s

es

y Pr

sit

er

Associate

sociate

ciate Prof

Professor

P

iv

Department of Information

formatio Science and Technology

Un

Engineering, Guindy Campus

Engi

Anna UnUniversity, Chennai

d

or

xf

O

KhW

Oxford University Press is a department of the University of Oxford.

It furthers the Universitys objective of excellence in research, scholarship,

and education by publishing worldwide. Oxford is a registered trade mark of

Oxford University Press in the UK and in certain other countries.

Published in India by

Oxford University Press

YMCA Library Building, 1 Jai Singh Road, New Delhi 110001, India

ed, stored

store in

a retrieval system, or transmitted, in any form or by any means,

ns, without

withou the

ss

prior permission in writing of Oxford University Press, or as expressly

pressly permitted

pe

by law, by licence, or under terms agreed with the appropriate

priate reprographics

iate repro

re

rights organization. Enquiries concerning reproduction outside the scope of the

above should be sent to the Rights Department, Oxfordo d University

Universi Press, at the

yP

address above.

k in any oth

sit other form

and you must impose this same condition

ndition on any

an acquirer.

ISBN-13: 978-0-19-809369-5

-0-19-809369-5

19-80936

er

ISBN-10:: 0-19-809369-1

-19-80936

iv

by Ideal

all Publishing

Publishin Solutions,

S Delhi

Un

Printed in India

ndia by Radha

Rad Press, New Delhi 110031

Third-party website

ite addresses

ebsite a dresses

dres mentioned in this book are provided

by Oxford University

versity Press

P ss in good faith and for information only.

d

disclaim any responsibility for the material contained therein.

or

xf

O

KhW

Features of the Book

C H A P T E R

Topical Coverage

Introducon to

Algorithms

1 C H A P T E R

The book provides

extensive coverage of

18 Basics of Computaonal

Complexity C H A P T E R

Data StructuresI 5 C H A P T E R

11 Greedy Algorithms

design techniques, Bad programmers worry about the code. Good programmers worry about data structures

followed by discussions

s

WZD C Hby Athe way...

Greed is all right, P T I think

E Rgreed is healthy. You can be greedy

13

es

d till f l d b t lf

Dynamic Programming

y Pr

sit

Treatment of Concepts

er

11.8.1 M S T

The history of an MST is as interesting as its concept. In 1926, Otaker

er Boruvka formulated

formula

form ^

the MST problem. A Polish mathematician, Vojtech Jarnik, described bed

d the probl

problem in 1929

iv

ndependen by Kruskal

in 1956. +ence, Kruskal rediscovered the problem. /ater it was as dened independently

iindep by are provided using

Un

Example 11.12 Consider the graph G shown sh wn in Fig.

Fig 11.11. Construct an MST for the

given graph G using Kruskals algorithm.

lgorithm is to sort all the edges and form an edge list,

6ROXWLRQ The rst step in Kruskals algorithm

d

11.21 are sorted and shown in Table 11.13.

or

<

Since the disjoint set data structure is used, ed, initiali

initialization takes at most (|V|) time. The time

xf

complexity of the algorithm depends ends on the numnumber of edges. As there are |E| edges, O(| E

|log| E |) time is required to sortt these edge

edges. The disjoint set takes at most 2| E | nd opera-

O

refore, th

tions and |V |1 operations. Therefore, the total complexity of Kruskals algorithm is at most

O(| E |log| E |) time.

Algoritm Presentaon

Step 1: Create a node x by allocating memory for it.

in two ways, that is, step- Step 2: Assign the required value to the item part of node x.

wise approach item(x) = value

Step 3: Set in the pointer to null.

next(x) = null

pseudocode approach Step 4: ReturnAlgorithm

the node x. create(L, x, value)

%% Input: List L and element x with 'value'

%% Output: Node x

Begin

understanding of the allot(x) %% Allot memory for node x with two eldsitem and next

item(x) = value

logic behind solving a next(x) = null

KhW

Historical Notes Box 1.1 Origin of the word algorithm

The word algorithm is derived from the name of a Persian Europe. He also introduced the simple step-b

Historical notes are mathematician, Abu Jafer Mohammed Ibn Musa al for addition, subtraction, multiplication, and

provided throughout the Khowarizmi, who lived sometime around 780850 AD. He his book. The word algebra has also been d

was from the town of Khowarazm, now in Uzbekistan. He the title of this book. When his book was t

Box 17.2 inGeorge

was a teacher of mathematics Bernard

Baghdad. Dantzig

He wrote a book Latin, his name was quoted as Algorismus

d *eorge Bernard Dantzig was born in 1914 at Portland, duality theory. He worked with Fulkerson an

Oregon, United States. His father became a professor of in formulating the travelling salesperson pro

mathematics at University of Maryland after World War linear programming and solved the TSP proble

II. Dantzigs biggest contribution is that he designed the 49 cities at that time. In 197, he was award

simplex method for solving LPPs. Apart from the simplex National Medal of Science, the highest honou

Q GLOSSARY Q

Agent A performer of an algorithm $OJRULWKPYHULFDWLRQ The process of provid

Algorist A person who is skilled in algorithm development cal proof that the algorithm works correctly f

ss

Algorithm A step-by-step procedure for solving a given Algorithm validation The process of chec

problem ness of an algorithm, this is done by givin given at the end of each

giv

Algorithm gap The difference between lower and upper it and checking its results with expected

SUMMARY

re

Q Q

chapter to help readers

An algorithm is a step-by-step procedure for solving a Algorithm verication is a process

es of provv

yP

given problem. ematical proof that the givenn algorithm

algorith

orit wo o

A computational problem is characterized by two for all instances of data.

factorsspecication of valid input and output param- A proof of an algorithm

sit m is said to exist

exis if t

er

iv

Eercises and

Un

1.2 at are the characteristics

What char of an algorithm" and algorithm validation"

Addional Prolems 1.3 S

Surveyy the Inte

Internet and list out at least ve algorithms 1.9 How is an algorithm validated and

h

that have huge impact on our daily lives. with an example.

E 4

1.4 a the stages of problem solving"

What are 1.10 State algorithm classications. Wha

EXERCISES Q

d

Q

1.1 Assume that there are two algorithms A and B for a complexities of A, B, and C are 3n

or

given problem P. The time complexity functions of algo- respectively. Assume that the input i

rithms A and B are, respectively, 5n and log2n. Which Assume that the machine executes

algorithm should be selected assuming that all other per second. How much time will algo

xf

pter

the end of every chapter conditions remain the same for both the problems" C take" Which algorithm will be the

h ADDITIONAL diPROBLEMh

Q Q

12 L f l h

er

to test the readers

O

1.1 John MacCormick had written a book titled Nine recognition, data compression, datab

conceptual knowledge ge Algorithms That Changed the Future: The Ingenious signatures, that changed the world.

Ideas that drive Todays Computers, Princeton University (a) What are these algorithms" Se

and also enhance their Press, Princeton, that had listed the nine wonderful and nd what these algorithms

algorithms, namely, search engine indexing, page rank, (b) Identify one more algorithm that

Q CROSSWORD Q

1 Crossword Puzzles

2

3

Crossword puzzles,

4 5

exercise at the end of each

7

8

9 10 readers to self-check their

11

KhW

W

Computers have become an integral part of our daily life in recent times. They have enormously impacted

our personal, professional, as well as social lives. Computers help us in tasks such as document editing,

Internet browsing, sending emails, making presentations, performing complex scientic computations,

social networking, and playing games. Industries and government ofces use computers effectively in

production, e-governance, and e-commerce. Considering the increasing demand of computers in society,

schools, colleges, and universities have included computer education in their curriculum, to help students

become skilled in programming and developing applications which can be used to solve various business,

scientic, and social problems.

Programming is a process of converting a given problem into an executable utable

able co

code for the computer. It

ss

involves understanding, analysis, and solving problems to create an algorithm. rithm.

hm. Veri

Verication of the algorithm,

coding of the algorithm in a specic programming language, testing, ing, debu

debug

debugging, and maintaining the

re

source code are also part of the programming process. Therefore, in n order to construct efcient programs,

yP

a ne understanding of algorithms is essential. An algorithm is a sset et of

o llogical instructions for solving a

given problem. It is expected to give correct results for valid alid inputs

input aand should be efcient, consuming

less computer resources. An algorithm is implemented ass a program using a programming language. A

sit

well-designed algorithm runs faster and consumes lesser sser com

compu

computer resources, namely time and space.

Therefore, expertise in programming is more related atedd to efc

efcie

efciency in problem-solving and effective de-

er

signing of algorithms, rather than developing codes with th the help of programming languages. Though

programming languages are important, theirr role ole is ju

just li

limited to the implementation of a well-designed

iv

algorithm. For this reason, algorithms aree a central th theme of computer study.

Un

than that of computers, dating back to 3000 BC. The

vilizat

lizat on w

ancient people of the Sumerian civilization wer

were aware of basic numeric computation like addition. A

uphrates

rates riv

Sumerian tablet found in the Euphrates river showed how to partition a given quantity of wheat in a way

d

specied quantity. Such tablets were also used by ancient Babylonians

or

ematicians

aticians oof this period such as Euclid, Al-Khwarizmi, Leonardo Pisano (also

(2000 to 1650 BC). Mathematicians

known as Fibonacci), and nd others produced

p procedures that provided the foundations of the concepts of

xf

velopment

elopmen in the eld of algorithms were due to the contributions of Gottfried

algorithms. Later developments

O

rt, and Alan Turing. The history of modern computers, however, only starts from

Leibnitz, David Hilbert,

the 1940s. Thus, algorithms have played a very important role in the development of modern computing.

The study of algorithms, called algorithmics, includes three aspectsalgorithm design, analysis, and

computational complexity of problems. Algorithm design is a creative activity. It includes various techniques

(such as divide-and-conquer, greedy approach, dynamic programming, backtracking) that help in producing

outputs at a faster pace by consuming lesser computer resources. Algorithm analysis is the estimation of

how much resource is required by the algorithm. Computational complexity deals with the analysis and

solvability of problems itself. A compulsory course on algorithms design and analysis is generally offered

to computer science and information technology students in most universities. The course aims to help

students create efcient algorithms in common engineering design situations and analyse the asymptotic

performance of algorithms by using the important algorithmic design paradigms and methods of analysis.

Design and Analysis of Algorithms is designed to serve as a textbook for the rst level course in algorithms

that discusses all the fundamental and necessary information related to the three important aspects of

KhW

Preface vii

algorithm study using minimal mathematics and lucid language. This book is suitable for undergradu-

ate students of computer science and engineering (CSE) and information technology (IT), as well as

for postgraduate students of computer applications. It is also useful for diploma courses, competitive

examinations (like GATE), and recruitment interviews for this subject.

The book begins with an introduction to algorithms and problem-solving concepts followed by an

introduction to algorithm writing and analysis of iterative and recursive algorithms. In-depth explana-

tions and designing techniques of various types of algorithms used for problem-solving such as the

brute force technique, divide-and conquer-technique, decrease-and-conquer strategy, greedy approach,

transform-and-conquer strategy, dynamic programming, branch-and-bound approach, and backtracking,

are provided in the book. Subsequent chapters of the book delve into the discussion of string algorithms,

iterative improvement, linear programming, computability theory, NP-hard problems, NP- completeness,

probability analysis, randomized algorithms, approximation algorithms, and parallel algorithms, with

the appendices throwing light on basic mathematics and proof techniques.

The various design techniques have been elucidated with the help off numer numerous problems, solved

numero

ss

examples, and illustrations (including schematics, tables, and cartoons).. The algor

algorithms are presented in

plain English (informal algorithm presentation) and pseudocode approach proach (formal algorithm presenta-

oach (for

re

tion) to make the book programming language-independent and easy-to-comprehend.

sy-to-com The book includes

a variety of chapter-end pedagogical features such as point-wise isee ssummary,

umm

umma glossary, review questions,

yP

exercises, and additional problems to help readers assimilate

tee and im implement

imp l the concepts learnt.

sit

KEY FEATURES

er

Provides simple and coherent explanations without ithout

out using

usin eexcessive theorems, proofs, and lemmas

Detailed coverage for topics such as greedy dynamic programming, transform-and-conquer

y approach, dy

iv

technique, decrease-and-conquer technique, que, linea programming, and randomized and approximation

linear pr

Un

algorithms

Dedicated chapters on backtracking ng and

ing branch-and-bound techniques, string matching algorithms,

nd bra

branc

and parallel algorithms

d

Simple and judicious presentationation of algorithms throughout the text in both informal and formal

entation

or

scussion oof their complexity analysis

Numerous review questions,

estions, exercises, and additional problems given at the end of each chapter to

ns, exe

xf

practise the concepts learnt

Includes glossary and poi point-wise summary at the end of each chapter to help readers quickly

O

rtan concepts

Provides historical notes on various topics and crossword puzzles at the end of each chapter to elicit

learning interest in students

The book consists of twenty chapters. A chapter-wise scheme of the book is presented here.

Chapter 1 provides an overview of algorithms. It introduces all the basic concepts of algorithms and

the fundamental stages of problem-solving. The chapter ends with the classication of algorithms.

Chapter 2 starts with the basic tools used for problem-solving. All the guidelines required for present-

ing the pseudocode and ow charts are provided along with many examples. The focus of this chapter

is to provide some practice on writing algorithms. The basics of recursion and algorithm correctness are

also covered in this chapter.

Chapter 3 covers the basics of algorithm complexity and analysis of iterative algorithms. Step count

and operation count methods used for analysing iterative algorithms are discussed in detail. Asymptotic

KhW

viii Preface

analysis is also discussed in detail. Finally, the chapter ends with the concept of analysing the efciency

of algorithms.

Chapter 4 discusses the analysis of recursive algorithms. It explains the basics of recurrence equations

along with the methods for solving them. Generating functions are also briey discussed as part of this chapter.

Data structures are an important component of algorithms. Chapter 5 deals with the fundamentals

concepts related to data structures such as stacks, queues, linear lists, and linked lists. Trees and graphs

are also explained in this chapter. Chapter 6 covers advanced data structures used for organizing

large amounts of dynamic data. Binary search trees and AVL trees used for organizing dictionaries

are discussed in this chapter, in addition to priority queues and heaps. Finally, a discussion on disjoint

sets and amortized analysis is provided.

Chapter 7 deals with the brute force techniques, which use no special logic but instead follow an

intuitive way of solving problems using the problem statement. Various problems such as sequential

search, bubble sort, and selection sort that can be solved using this approachh are a explained. Some basic

computational geometry problems such as closest-pair and convex hull aree also covered. The chapter

ss

ends with the discussions on exhaustive searching problems such as 15-puzzle 5-puzzle

puzzle pproblem, 8-queen prob-

lem, magic square problem, knapsack problem, container loading problem, and assignment problem.

blem, an

re

Chapter 8 discusses the divide-and-conquer design paradigm. Important problems such as quicksort,

merge sort, nding maximum and minimum, multiplication off lon long integers, Strassen matrix multiplica-

ong int

integ

yP

tion, tiling problem, closest-pair problem, and convex hull are solv solved

solved using this technique. The chapter

ends with a discussion of the Fourier transform problem. m.

sit

Chapter 9 explains the decrease-and-conquer technique, hnique, which is also known as the incremental or

ique, w

inductive approach. Examples problems such as insertion nsertion sor ttopological sort, generating permutations

rtion sort,

er

and subsets, binary search, fake coin detection, andd Russian

Russi peasant multiplication problem are used to

illustrate the decrease by constant and constant methods. Finally, discussions on interpolation search,

nt factor m

metho

iv

selection, and nding the median problems ms using ththe ddecrease by variable factor method are provided.

Un

Chapter 10 deals with timespace tradeoffs. The selection of one type of efciency over the other

radeoffs. T

Th

and problems related to linear sorting ngg aand Hashing are discussed. The chapter ends with a discussion on

d Has

Hashin

B-trees and their operations.

d

eedy approach concept. This chapter discusses important problems such as

dy approa

or

duling, knapsack problem, optimal storage of tapes, Huffman code, minimum

uling, kna

spanning tree algorithms, Dijkstras shortest path algorithm to illustrate the greedy approach.

ms, and Dijk

xf

Chapter 12 discusses

sses transform-and-conquer approach and its three basic techniquesinstance

ses the tran

tra

entatio change, and problem reduction. Problems such as Gaussian elimination,

entation

simplication, representation

O

decomposition methods, nding determinant and matrix inverses are discussed. Heap sort, Horners

method, binary exponentiation algorithm, and reduction problems are also covered in this chapter.

Chapter 13 deals with dynamic programming. Important example problems such as Fibonacci prob-

lem, binomial coefcient multistage graph, Graph algorithms, Floyd-Warshall Algorithm, BellmanFord

algorithms, travelling salesman problem, chain matrix multiplication, knapsack problem, and optimal

binary search tree problem are discussed to illustrate the dynamic programming concept. Finally, the

chapter concludes with the ow-shop scheduling algorithms.

Chapter 14 discusses backtracking algorithms. This chapter covers important problems such as N-queen

problem, Hamiltonian circuit problem, sum of subsets, vertex colouring problem, graph colouring prob-

lems, Graham scan, and generating permutations.

Chapter 15 explains branch-and-bound techniques. Search techniques using this concept are discussed.

Important problems such as assignment problem and 15-puzzle are covered and the chapter ends with a

discussion on traveling salesperson and knapsack problems.

Chapter 16 deals with the string algorithms. Some basic string algorithms such as nding the length

of strings, nding substrings, concatenation of two strings, longest common sequence, and pattern

KhW

Preface ix

are discussed in this chapter. Finally, approximate string matching algorithm is discussed.

Chapter 17 discusses the iterative approach and basics of linear programming. The linear programming

formulation of a problem and simplex method is discussed in this chapter. Minimization problem, prin-

ciple of duality, and max-ow problems are also explained. Finally, matching algorithms are considered

for better understanding of computational complexity.

Chapter 18 explains the basics of computational complexity and the upper and lower bound theory.

Decision problems, complexity classes, and reduction concepts are also discussed. This chapter also

covers theory of NP-complete problems and examples for proving NP-completeness.

Chapter 19 covers the basic concepts and types of both randomized and approximation algorithms.

Randomized algorithms are illustrated through examples such as hiring problem, primality testing, com-

parison of strings, and randomized quicksort. Approximation algorithms are illustrated through examples

based on heuristic, greedy, linear, and dynamic programming approaches.

Chapter 20 begins with an introduction to parallel processing and classication

classic

ssic of parallel sys-

ss

tems. It then discusses the fundamentals of parallel algorithms and parallel rallel ran

random access machine

(PRAM) model. The concept of parallelism is illustrated through examples xamples related to parallel search-

mples re

re

ing, parallel sorting, and graph and matrix multiplication problems. lems.

There are two appendices in this book. Appendix A explains thee ba

ns th basics of mathematics such as sets,

basic

yP

series and sequences, relations, functions, matrix algebra, and probability

d probab

probabil i that are necessary for algorithm

study. Appendix B deals with mathematical logic and proof techniques.

ooff techniqu

technique

sit

ONLINE RESOURCES

er

ompanied

mpanied by online resources that are available at http://

iv

nalysis-Alg

nalysis-Algori

oupinheonline.com/book/sridhar-Design-Analysis-Algorithms/9780198093695. The content for the online

resources are as follows:

Un

For Instructors

d

PowerPoint slides

or

Solutions manual

xf

For Students

Answers to the crossword

ssword

sword ppuzzles

O

ACKNOWLEDGEMENTS

This book would not have been possible without the help and encouragement of many friends, colleagues,

and well-wishers. I thank all my students who motivated me by asking interesting questions related to

the subject. I acknowledge the assistance of my friend, P. Kanniappan, in typing the manuscripts and for

all his advice. I also express my gratitude to all my colleagues at the departments of Computer Science

and Engineering and Information Science and Technology, Anna 8niversity and National Institute of

Technology, Tiruchirapalli for reviewing my manuscripts and providing constructive suggestions for

improvement. I acknowledge all my reviewers whose feedback has helped making this book better. I

am also very thankful to the editorial team at Oxford 8niversity Press, India for providing valuable as-

sistance. My sincere thanks are also due to my family members, especially my wife, Dr N. Vasanthy, my

mother, Mrs Parameswari, my mother-in-law, Mrs Renuga, and my children, Shobika and Shreevarshika,

for their constant support and encouragement during the development of this book.

S. Sridhar

KhW

Brief Contents

Features of the Book iv

Preface vi

Detailed Contents xi

1 Introduction to Algorithms 1

2 Basics of Algorithm Writing 22

3 Basics of Algorithm Analysis 58

4 Mathematical Analysis of Recursive Algorithms 98

ss

5 Data StructuresI 141

re

6 Data StructuresII 194

yP

7 Brute Force Approaches 231

8 Divide-and-conquer Approach 262

sit

9 Decrease-and-conquer Approach 309

er

10 TimeSpace Tradeoffs 342

iv

Un

13 Dynamic Programming 455

d

14 Backtracking 517

or

15 Branch-and-bound Technique

chniqu 543

xf

16 String Algorithms

mss 569

O

18 Basics of Computational Complexity 638

19 Randomized and Approximation Algorithms 663

20 Parallel Algorithms 705

Appendix BProof Techniques 752

Bibliography 763

Index 766

KhW

Detailed Contents

Features of the Book iv

Preface vi

Brief Contents x

1.1 Introduction 1 2.3.1 Flowcharts 32

1.2 Need for Algorithmic Thinking 1 2.4 Basics of Recursion 42

1.3 Overview of Algorithms 3 2.5 Recursive Algorithms 43

1.3.1 Computational Problems, rre

2.6 Algorithm Correctness 52

Instance, and Size 5

ss

1.4 Need for Algorithm Efciency 7 3 Basics of Algorithm

gorithm

rithm AAn

Analysis 58

1.5 Fundamental Stages of Problem 3.1 Basics

cs of Algor

Algorithm Complexity 58

re

Solving 9 3.2 Introduction

roductio to Time complexity

roduction 60

yP

1.5.1 Understanding the Problem 9 3.2.

3.2.1

2 Ran

R

Random Access Machine 60

1.5.2 Planning an Algorithm 9 3.33 Anal

Analys

Analysis of Iterative Algorithms 62

1.5.3 Designing an Algorithm 11 3.3.1 Measuring Input Size 63

sit

1.5.4 Validating and Verifying an 3.3.2 Measuring Running Time

3.3 63

3.3.3 Best-, Worst-, and

er

Algorithm 11

1.5.5 Analysing an Algorithm 12 Average-case Complexity 71

iv

Un

Analysis 14 3.4.2 Comparison Framework 74

1.5.7 Post (or Postmortem)

em) 3.5 Asymptotic Analysis 76

d

or

gorithms 15 3.5.2 Asymptotic Rules 85

1.6.1 Based onn Implement

Implemen

Implementation 15 3.5.3 Asymptotic Complexity Classes 87

xf

O

Specialization 17 Visualization 88

1.6.4 Based on Tractability 18 3.7.1 Experimental Purpose 89

3.7.2 Statistical Tests 90

2 Basics of Algorithm Wring 22

2.1 Tools for Problem-solving 22 Mathemacal Analysis of Recursive

2.1.1 Stepwise 5enement or Algorithms 98

Top-down design 23 4.1 Introduction to Recurrence

2.1.2 Bottom-up Approach 24 Equations 98

2.1.3 Structured Programming 25 4.1.1 Linear Recurrences 99

2.2 Algorithm Specications 26 4.1.2 Non-linear Recurrences 101

2.2.1 Guidelines for Writing 4.2 Formulation of Recurrence

Algorithms 27 Equations 103

KhW

xii Detailed Contents

Recurrence Equations 106 6.1 Introduction to Dictionary 194

4.3.1 Guess-and-verify Method 106 6.1.1 Introduction to Binary

4.3.2 Substitution Method 109 Search Tree 194

4.3.3 Recurrence-tree Method 112 6.1.2 AVL Trees 200

4.3.4 Difference Method 117 6.2 Priority Queues and Heaps 206

4.4 Solving Recurrence Equations 6.2.1 Binary Heaps 207

8sing Polynomial Reduction 118 6.2.2 Binomial Heaps 212

4.4.1 Solving Homogeneous 6.2.3 Fibonacci heap 218

Equations 118 6.3 Disjoint Sets 221

4.4.2 Solving Non-homogeneous 6.3.1 Representation and

Equations 122 Operations

ion 221

4.5 Generating Functions 123 6.4 Amortized d Analys

Analysis 224

ss

4.5.1 Properties of Generating 6.4.1 Aggregate

gregate MMethod 225

Functions 124 2 Accounti

6.4.2 Account

Accounting Method 226

re

4.6 Divide-and-conquer Recurrences 127 4.3 Poten

6.4.3 Pote

Potential Method 226

yP

4.6.1 Master Theorem 127

4.6.2 Transformations 133 7 Brute Fo

Forc

Force Approaches 231

4.6.3 Conditional Asymptotics 135 7.1 Intro

Introduction 231

sit

77.1.1 Advantages and

er

5 Data StructuresI 141

1 Disadvantages of Brute

5.1 Data Structures and Algorithms 141 Force Method 232

iv

Un

143 7.2.1 Analysis of Recursion

5.2.2 Linked Lists 146

14 Programs 234

5.3 Stacks 152 7.2.2 Recursive Form of Linear

5.3.1 Representationn of and Search Algorithm 235

Operationss onn Stacks 152 7.3 Sorting Problem 236

5.4 Queues 154 7.3.1 Classication of Sorting

5.4.1 Queuee Represe

Represen

Representation 154 Algorithms 237

5.5 Trees 157 7.3.2 Properties of Sorting

5.5.1 Tree Terminologies 157 Algorithms 237

5.5.2 Classication of Trees 159 7.3.3 Bubble Sort 238

5.5.3 Binary Tree Representation 161 7.3.4 Selection Sort 242

5.5.4 Binary Tree Operations 163 7.4 Computational Geometry Problems 245

5.6 Graphs 167 7.4.1 Closest-pair Problem 245

5.6.1 Terminologies and 7.4.2 Convex Hull Problem 247

Types of Graphs 168 7.5 Exhaustive Searching 249

5.6.2 Graph Representation 172 7.5.1 15-puzzle Problem 250

5.6.3 Graph Traversal 174 7.5.2 8-queen Problem 251

5.6.4 Elementary Graph Algorithms 178 7.5.3 Magic Squares 252

5.6.5 Spanning Tree and Minimum-cost 7.5.4 Container Loading Problem 253

Spanning Tree 186 7.5.6 Assignment Problem 256

KhW

Detailed Contents xiii

8.1 Introduction 262 9.4.2 Selection and Ordered

8.1.1 Recurrence Equation for Statistics 334

Divide and Conquer 263 9.4.3 Finding Median 336

8.1.2 Advantages and Disadvantages

of Divide-and-conquer 10 TimeSpace Tradeos 342

Paradigm 264 10.1 Introduction to TimeSpace

8.2 Merge Sort 264 Tradeoffs 342

8.3 Quicksort 269 10.2 Linear Sorting 342

8.3.1 Partitioning Algorithms 270 10.2.1 Counting Sort 342

8.3.2 Variants of Quicksort 275 10.2.2 Bucket Sort 347

8.4 Finding Maximum and Minimum 10.2.3 Radix So

Sort 349

Elements 277 10.3 Hashing andd Hash Tables 351

ss

8.5 Multiplication of Long Integers 280 10.3.1 Properties

roperties of Hash

8.6 Strassen Matrix Multiplication 284 Functions

Functi

Functio 353

re

8.7 Tiling Problem 288 10.3.2

0.3.2 Hash

Ha Table Operations 354

yP

8.8 Closest-pair Problem 291 10.3.3

0 3 Collision

10.3 C 356

8.8.1 Using Divide-and-conquer 10.4 B-tree

B-tr

B-trees 359

Method 291 10.4.1

10

10.4

4 B-tree Balancing

sit

8.9 Convex Hull 293 Operations 361

er

8.9.1 Quickhull 2933 10.4.2 B-tree Operations 363

8.9.2 Merge Hull 295

95

iv

1 372

Un

ion

on 296

29 11.1 Introduction to Greedy Approach 372

8.10.2 Application of Fourier

rier

er 11.1.1 Components of Greedy

Transform 2988 Algorithms 373

d

Transform 302 11.2 Suitability of Greedy Approach 374

or

11.4 Scheduling Problems 376

xf

9 Decrease-and-conquer

nquer App

Approach 309

9.1 Introduction 309 11.4.1 Scheduling without

O

onst Method 311 Deadline 377

9.2.1 Insertion Sort 311 11.4.2 Scheduling with Deadline 379

9.2.2 Topological Sort 315 11.4.3 Activity Selection Problem 382

9.2.3 Generating Permutations 321 11.5 Knapsack Problem 384

9.2.4 Generating Subsets 323 11.6 Optimal Storage of Tapes 388

9.3 Decrease by Constant Factor 11.7 Optimal Tree Problems 390

Method 325 11.7.1 Optimal Merge 390

9.3.1 Binary Search 326 11.7.2 Huffman Coding 393

9.3.2 Fake Coin Detection 330 11.7.3 Tree Vertex Splitting Problem 398

9.3.3 Russian Peasant 11.8 Optimal Graph Problems 401

Multiplication Problem 331 11.8.1 Minimum Spanning Trees 401

9.4 Decrease by Variable 11.8.2 Single-source Shortest-path

Factor Method 332 Problems 407

KhW

xiv Detailed Contents

12.1 Introduction to Transform and Shortest-path Algorithm 478

Conquer 417 13.6.1 Shortest-path

12.2 Introduction to Instance Reconstruction 481

Simplication 418 13.7 BellmanFord Algorithm 481

12.3 Matrix Operations 419 13.8 Travelling Salesperson Problem 486

12.3.1 Gaussian Elimination 13.9 Chain Matrix Multiplication 488

Method 420 13.9.1 Dynamic Programming

12.3.2 LU Decomposition 427 Approach for Solving Chain

12.3.3 Crouts Method of Matrix Multiplication

Decomposition 434 Problem 490

12.3.4 Finding Matrix Inverse 436 13.10 Knapsack Problem

Pr 496

12.3.5 Finding Matrix 13.11 Optimall Binary Search Trees 500

ss

Determinant 439 13.11.1

1 Brute FForce Approach

12.4 Change of Representation 441 for Constructing

C Optimal

re

12.4.1 Heap Sort 441 B

BSTs 500

yP

12.4.2 Polynomial Evaluation 13.11.2

3.11 Dynamic Programming

13.11.2

Using Horners Method 445 Approach for Constructing

12.4.3 Binary Exponentiation 447 Optimal BSTs 502

sit

12.5 Problem Reduction 450 13.1 Flow-shop Scheduling Problem 507

13.12

13.12.1 Single-machine

er

iv

13.1.1 Components of Dynamicc

Un

Problem 509

Programming 456

13.1.2 Characteristics of Dynamic

Dynam

ynam 14 Backtracking 517

14.1 Introduction 517

d

Programmingg 458

14.2 Basics of Backtracking 518

or

m 460

13.3 Computing Binomial Coefcients 463

omial Co 14.3 N-queen Problem 522

xf

Pro 466 14.3.1 State Space of 4-queen

Problem 523

O

13.4.1 Forward

ard Computation

Co

Procedure 467 14.4 Sum of Subsets 525

13.4.2 Backward Computation 14.5 Vertex Colouring Problem 527

Procedure 471 14.6 Hamiltonian Circuit Problem 531

13.5 Transitive Closure and Warshall 14.6.1 Promising (or Bounding

Algorithm 472 Function) for Hamiltonian

13.5.1 Finding Transitive Closure Using Problem 531

Brute Force Approach 473 14.7 Generating Permutation 534

13.5.2 Finding Transitive Closure Using 14.8 Graham Scan 536

Warshall Algorithm 473 15 Branch-and-ound Technique 543

13.5.3 Alternative Method to 15.1 Introduction 543

Warshall Algorithm for 15.2 Search Techniques for Branch-

Finding Transitive Closure 476 and-bound Technique 545

KhW

Detailed Contents xv

bound algorithm 17.9 Bipartite Matching Problem 628

FIFOBB 545 17.10 Stable Marriage Problem 630

15.2.2 LIFO with Branch and

Bound 547

18 Basics of Computaonal

15.2.3 Least Cost with Branch

Complexity 638

and Bound 547

18.1 Introduction to Computational

15.3 15-puzzle Game 548

Complexity 638

15.4 Assignment Problem 552

18.2 Algorithm Complexity,

15.5 Traveling Salesperson Problem 559

8pper and Lower Bound Theory 639

15.6 Knapsack Problem 562

18.2.1 Upper and Lower Bounds 640

18.3 Decision Problems

oble

ble and Turing

16 String Algorithms 569 Machine 644

ss

16.1 Introduction to String 18.4 Complexityxity Classes

plexity Cla 647

Processing 569

re

18.4.1

4.1 Class P 648

16.2 Basic String Algorithms 571 118.4.2

.4.2 NP Class 648

yP

16.2.1 Length of Strings 571 18.5

.5 Th

Theo

Theoryry of NP-complete

16.2.2 Concatenation of Problems

Prob

Proble 650

Two Strings 571

sit

18.6 Reductions

Red 651

16.2.3 Finding Substrings 572 18.6.1 Turing Reduction 651

16.3 Longest Common Subsequences 573

er

18.6.2 Karp Reduction 652

16.4 nave String Matching 18.7 Satisability Problem and

iv

Un

16.5 Pattern Matching 8sing Finitee 18.8 Example Problems for Proving

Automata 578

5 NP-completeness 655

16.6 RabinKarp Algorithm m 580

5 18.8.1 SAT is NP-complete 655

d

or

thm

m 588 NP-complete 656

16.9 BoyerMooree String Matching

Ma

M

xf

Algorithm 590 is NP-complete 657

O

Stri Matching 594 18.8.4 Sum of Subsets

(from 3-CNF-SAT) 658

17 Iterave Improvement and

Linear Programming 601 19 Randomized and Approximaon

17.1 Introduction to Iterative Algorithms 663

Improvement 601 19.1 Dealing with NP-hard

17.2 Linear Programming 601 Problems 663

17.3 Formulation of LPPs 603 19.2 Introduction to Randomized

17.4 Graphical Method for Algorithms 664

Solving LPPs 606 19.2.1 Generation of Random

17.5 Simplex Method 611 Numbers 666

17.6 Minimization Problems 615 19.2.2 Types of Randomized

17.7 Principle of Duality 619 Algorithms 669

KhW

xvi Detailed Contents

Algorithms 669 20.2.1 Flynn Classication 706

19.3.1 Hiring Problem 669 20.2.2 Address-space (or Memory

19.3.2 Primality Testing Algorithm 672 Mechanism-based)

19.3.3 Comparing Strings Using Classication 708

Randomization Algorithm 674 20.2.3 Classication Based on

19.3.4 Randomized Quicksort 675 Interconnection Networks 710

19.4 Introduction to Approximation 20.3 Introduction to PRAM Model 711

Algorithms 678 20.4 Parallel Algorithm Specications

19.5 Types of Approximation and Analysis 712

Algorithms 679 20.4.1 Parallel Algorithm

19.6 Examples of Approximation Analysis

si 714

Algorithms 680 20.5 Simple Parallel

arallel

allel A

Algorithms 716

ss

19.6.1 Heuristic-based 20.5.1 Prex

rex Computation

Com 716

Approximation Algorithms 681 20.5.2

5.2 List Ranking

Ra 718

re

19.6.2 Greedy Approximation 0.5.3 Euler

20.5.3 Eu Tour

Eule 719

yP

Algorithms 688 20.6

6 Pa

P

Parallel

rall Searching and Parallel

rallel

19.6.3 Approximation Algorithm Sorting

Sor

Sorti 720

Design Using Linear 20.6.1

20

20.66 Parallel Searching 720

sit

Programming 693 220.6.2 OddEven Swap Sort 722

19.6.4 Designing Approximation 20.6.3 Parallel MergeSplit

er

iv

Un

20 Parallel Algorithms 705 Multiplication 726

20.1 Introduction to Parallel

el Process

Processing

ng 705 20.7.2 Parallel Graph Algorithms 729

d

or

Appendix AMathematical

ical Basics 735

xf

echniques 752

Bibliography 763

O

Index 766

KhW

C H A P T E R

Introducon to

Algorithms

1

Ideas are the beginning point of all fortunes.

Napolean Hill

1.1 INTRODUCTION

ss

Learning Oecves

Computers are powerful tools of computing. ing.

g. One ca

cannot ignore the impact of

re

This chapter introduces

the basics of algorithms.

computers on our modern life. We usee computecomputers for personal needs such as

computer

typing documents, browsing the Internet,

ntern ssending emails, playing computer

ernet, sen

yP

All important denitions

and concepts related to games, performing numeric calculations,

alculation and so on. Industries and govern-

alculations,

the study of algorithms are ments use computers much effectively to perform complicated tasks

h more effec

effecti

sit

the focus of this chapter. to improve productivity and efciency. Applications of computer systems in

d efcie

efciency

The reader would be

airline reservation, video

deo surveillance, biometric recognition, e-governance, and

o surveil

surveillanc

er

familiar with the following

concepts by the end of e-commerce are all lll examples of their usefulness in improving efciency and

iv

this chapter: productivity. The increasing iimportance of computers in our lives has prompted

he increasin

schools and universities tto introduce computer science as an integral part of

d universitie

Un

Basic terminologies of

algorithm study our modernern education.

dern educat Informally, everyone is expected to handle computers to

Need for algorithms accomplish

plish certain

mplish ce ain basic tasks. This knowledge of using computers to perform

d

Characteristics of our

ur day-to-day

day-to-da activities is often called computational thinking. Computational

or

algorithms

thinking isi a necessity to survive in this modern world. However, computer

Stages of a problem-

science

scien professionals are expected to accomplish much more than acquiring

xf

solving process

Need for efcient this

th basic skill of computer usage. They are required to write specic computer

O

Classication of program is slightly more complicated than merely learning to use computers,

algorithms as it requires algorithmic thinking. Algorithmic thinking is an important

analytical skill that is required for writing effective programs in order to solve

given problems. Algorithmic thinking is not conned to computer science only,

but it spreads through all disciplines of study. Computer science is a domain

where the skills of algorithmic thinking are taught to the aspiring computer

professionals for solving computational problems.

Norman E. Gibbs and Allen B. Tucker1 proposed a denition for computer

science that captures the core truth of computer science study. According to

1

Gibbs, Norman E., Allen B. Tucker, A Model Curriculum for a Liberal Arts Degree

in Computer Science, Communications of ACM, Vol. 29, Issue 3, 1986.

KhW

2 Design and Analysis of Algorithms

them, Computer science is the systematic study of algorithms and data structures, specically

their formal properties, their mechanical and linguistic realizations, and their applications.

Hence, there cannot be any dispute regarding the fact that study of algorithms is the central

theme of computer science.

Let us elaborate this denition further. The formal and mathematical properties of

algorithms include the study of algorithm correctness, algorithm design, and algorithm

analysis for understanding the behaviour of algorithms. Hardware realizations include

the study of computer hardware, which is necessary to run the algorithms in the form of

programs. Linguistic realizations include the study of programming languages and their

design, translators such as interpreters and compilers, and system software tools such as

linkers and loaders, so that the algorithms can be executed by hardware in the form of

programs. Applications of algorithms include the study of design es and development of

efcient software packages and software tools so that these se algor

algorithms can be used to

ss

solve specic problems.

Thus, computer scientists consider the study of algorithms

rithms the core theme of computer

hms as th

re

science. The art of designing, implementing, and analysing algorithms is called algorithmics.

ysing alg

algo

yP

Algorithmics is a general word that comprises alll aspe

aspects

p cts of the study of algorithms. Let us

now attempt to dene algorithms formally. An algorithm is i a set of unambiguous instructions

or procedures used for solving a given problem em to provide

blem provi correct and expected outputs for

prov

sit

all valid and legal input data (refer to Box

ox 1.1).

An algorithm can also be referred d by othe terminologies such as recipe, prescription,

other te

er

dure. 1 provides a brief history of algorithms.

iv

rescriptions of

rescription o how a computer should carry out instructions

Un

to solve problems. Hence, algorithms ca can be visualized as strategies for solving a problem.

One cannot write a programgram given problem without the necessary analytical skills or a

ram for a giv

strategy for solving the proble

problem.

m TThus, the knowledge of how to solve a problem is called

d

algorithmic thinking.

king

g.

or

mpare program

pare a pr

pro construction with the construction of a house. The blueprint

of the housesee incorporating

incorpora

incorpor all the planning necessary for the construction of the house can

xf

be visualized

zed as an algorithm. A computer program is only an algorithm expressed using a

ized

O

ng la

using a programming language, the ability to conceive a strategy or apply analytical skills

are important for solving a problem or constructing a program.

The word algorithm is derived from the name of a Persian Europe. He also introduced the simple step-by-step rules

mathematician, Abu Jafer Mohammed Ibn Musa al for addition, subtraction, multiplication, and division in

Khowarizmi, who lived sometime around 780850 AD. He his book. The word algebra has also been derived from

was from the town of Khowarazm, now in Uzbekistan. He the title of this book. When his book was translated to

was a teacher of mathematics in Baghdad. He wrote a book Latin, his name was quoted as Algorismus from which

titled Kitab al Jabr wal Muqabala (Rules of Restoration the word algorithm emerged. Algorithm as a word thus

and Reduction) and Algoritmi Numero Indorum where became famous for referring to procedures that are used

he introduced the old ArabicIndian number systems to by computers for solving problems.

KhW

Introduction to Algorithms 3

The history of algorithms dates back to ancient civilizations Alan Turings Turing machines developed in 193 play

that used numbers. Ancient civilizations such as Egyptian, a very important role in computational complexity theory.

Indian, Greek, and Chinese were known to have used This Turing machine and Alonzo Churchs lambda calcu-

algorithms or procedures to carry out tasks like simple lus formed the basis for formalization of the complexity

calculations. This is in contrast to the history of modern theory. Thus, the history of algorithms is much richer than

computers that were designed in the 1940s only. Thus, that of computers.

the history of algorithms is much older and more exciting. The history of computing is another interesting story.

Euclid, who lived in ancient Greece around 400300 BC, Abacus and other mechanical devices used for computing

is credited with writing the rst algorithm in history for com- have been utilized at various stages of human history. A real

puting the greatest common divisor (GCD). Archimedes attempt was made in the 17th century by Leibniz, a German

created an algorithm for the approximation of the number mathematician and philosopher. He invented innitesimal

pi. Eratosthenes introduced to the world the algorithm for calculus, generating the idea of computers.

co In 1822, Charles

nding prime numbers. Averroes (1121198) also used Babbage introduced the difference

ference engine, which required

algorithmic methods for calculations. changing of gears manually ally to perform

nually perf calculations. These

A number of inuences led to the formalization of the ideas nally led to the invention of computers in the 1940s.

theory of algorithms. Some of the noteworthy developments In 1942, the US S government

governmen designed a machine called

include the introduction of Boolean algebra by George ENIAC (electronic

ctro ic numerical

numer integrator and computer). This

Boole (1847) and set theory by Gottlob Frege (1879). Set was succeeded

ceeded

eded b byy EDVAC

E (electronic discrete variable

theory has become a foundation of modern mathemat- automatic

atic computer)

matic compute in 1951. In 1952, IBM introduced

compu

ics. The concept of recursion formulated by Kurt Gdel, its mainframe

ainframe computer.

com Developments in the hardware

and contributions of Giuseppe Peano (1888), Alfred North domain

main and de declining costs have shifted computer usage

Whitehead, and Bertrand Russell in mathematical logic from big research

res and military environments to homes and

led to the formalization of algorithm as an independent dent educational

educ

educatio institutions. In 1981, IBM personal computers

eld of study. were

w introduced for home and ofce use.

Un

MSS

d

neric and nnot limited to computers alone (refer to Box 1.2). We perform

or

many algorithms

ms in our daddaily life unknowingly. Consider a few of our daily activities, some

of which are listed

ted as follows:

fo

xf

1. Searching

ng for a specic book

ing

O

2. Arranging bo

bbooks based on titles

3. Using a recipe to cook a new dish

4. Packing items in a suitcase

5. Scheduling daily activities

6. Finding the shortest path to a friends house

7. Searching for a document on the Internet

8. Preparing a CD of compressed personal data

9. Sending messages via email or SMS

We perform many tasks without being aware of their inherent algorithms. For example,

consider the activity of searching a word in a dictionary. How do we search? It can be noted

that indexing the words in a dictionary reduces the effort of searching signicantly. We just

open the book, compare the word with the index given, and accordingly decide on the por-

tions of the books to be searched for nding the meaning of that particular word. This kind

KhW

4 Design and Analysis of Algorithms

Agent

formally known as an algorithm.

The environment of a typical algorithm is shown in

Valid Fig. 1.1. Here, agent, which can be a human or a computer,

Algorithm Output

input is the performer.

Fig. 1.1 Typical algorithm To illustrate further, let us consider the example of tea

preparation. The hardware used here includes cooking

utensils a heater, and a person. To make tea, one needs to have the following ingredients

water, tea powder, and sugar. These ingredients are analogous to the inputs of an algorithm.

Here, preparing a cup of tea is called a process. The output of this process is, in this case, tea.

This corresponds to the output of an algorithm. The procedure for preparing tea is as follows:

1. Put tea powder in a cup. 4. Pour milk.

2. Boil the water and pour it into the cup. 5. Add sugar if necessary.

ecessary.

essary

ss

3. Filter it. 6. Pour the teaa into cup.

nto a cup

re

This kind of a procedure can be called an algorithm. hm.. It can be noted that an algorithm

hm

consists of step-by-step instructions that are required accomplish a certain task.

ed to accom

yP

Humans often perform such procedures intuitively vely or eeven mechanically without spending

ively

much conscious thought, and hence, they label sit actions as habitual activity. Many of our

ell such acti

action

day-to-day activities are not very efcient.. However,

However al algorithms that are meant for computers

represent a different case altogether. Computer procedures should be efcient as computer

mputer ppro

er

resources are scarce. Hence, much ought is ggiven for writing computer procedures that

h thought

can solve problems. Problems can classied into two types: computational problems and

an be clas

classie

iv

non-computational problems.

Un

is characterized by twoo factors:

actor : (a)

factor (a the

t formalization of all legal inputs and expected outputs

of a given problem m andd (b) the characterization of the relationship between problem output

d

and input. Thus, algorithm is expected to give the expected output for all legal inputs.

s, an algori

algorit

or

If an algorithm

hm yields tthe correct output for a legal input, then it is called an algorithmic

xf

solution.

Non-computational

omputati

mputati problems cannot be solved by a computer system. This classication

O

un differences between computers and humans in solving problems.

Computers are more effective than humans in performing calculations and can crunch num-

bers in a fraction of seconds with more precision and efciency. However, humans outscore

computers in recognition. The ability of recognizing an object by humans is much better than

that by machines. Recognition of an object by computer systems requires lots of programming

involving images and concepts of image processing. In addition, some tasks are plainly not

possible to be carried out by computer systems. Can a computer offer its opinion or show

emotion like humans? To put it simply, problems involving more intellectual complexity are

much more difcult to solve by computers as computer systems lack intelligence.

Thus, developing algorithms to make computers more intelligent and make them perform

tasks like humans becomes much more crucial and challenging. Developing algorithms for

computers is both an art and a science. Algorithm design is an art that involves a lot of creative

ideas, novelty, and even adventurous strategies and knowledge. Algorithm design is also a

science because its construction usually involves the application of some set of principles. The

KhW

Introduction to Algorithms 5

review and knowledge of these principles can facilitate the development of better algorithms,

based on sound mathematical and scientic principles.

One encounters many types of computational problems in the domain of computer science.

Some of the problems are as follows:

Structuring problems In structuring problems, the input is restructured based on certain

conditions or properties. For example, sorting a list in an ascending or a descending order is

a structuring problem.

Search problems A search problem involves searching for a target in a list of all possibili-

ties. All potential solutions may be generated and represented in ththe form of a list or graph.

n problem

Based on a property or condition, the best solution for the given roblem is searched. Puzzles

ss

are good examples of search problems where the target or goal oal is sea

searched in a huge list of

possible solutions.

re

Construction problems These problems involvee th

he const

the construction of solutions based on

yP

the constraints associated with the problem.

Decision problems Decision problems are re yes/no typ

type of problems where the algorithm

sit

output is restricted to answering yes or no. Let us assu

aassume that a road network map of a city

ny road conne

is given. The problem, say, Is there any con

connectivity between two cities, say Hyderabad

er

ecision

cision pro

and Chennai?, can be called as a decision proble

problem as the output of this algorithm is restricted

iv

Un

Optimization problems Optimization

that are often encounteredred in compu science domain. The decision problem about road con-

mpute

computer

nectivity between Chennai nnai and Hyderabad can also be posed as an optimization problem as

follows: What is the shortes

shortest distance between Chennai and Hyderabad? This is an optimiza-

tion problem, as thehe prob

probl

problem involves nding the shortest path. Thus, optimization problems

ertain

rtain objec

involve a certain object

objective function that is typically of the following form: maximize (say prot)

or minimizezee (say eeffort) based on a set of constraints that are associated with the problem.

ro

Once the problem is recognized, the input and output of an algorithm should be identied.

Consider, for example, a problem of nding the factorial of a number. The factorial of a positive

number N can be written as follows: N! = N (N 1) (N 2) . . . 1. A valid input can be

called an instance of a problem. For example, factorial of a negative number is not possible.

Therefore, all valid positive integers {0, 1, 2, . . .} can serve as inputs and every legal input is

called an instance. All possible inputs of a problem are often called a domain of the input data.

The input should be encoded in a suitable form so that computers can process it. The number

of binary bits used to represent the given input, say N, is called the input size. Input size is

important, as a larger input size consumes more computer time and space.

The core question still remains to be answered: how to solve a given problem? To illustrate

problem solving, let us consider a simple problem of counting the number of students in a

tuition centre who have passed or failed a test, assuming that the pass mark is 50. Details of

the students such as their registration number, name, and more importantly, marks obtained,

which are necessary for the given problem, are shown in Table 1.1.

KhW

6 Design and Analysis of Algorithms

1 Abraham 80

2 Beena 30

3 Chander 83

4 David 23

5 Elizabeth 90

Fauzia 78

The class strength of the tuition centre is 6. The pass mark of the course is given as 50. How do

we manually solve this problem? First, we will read the student marks marks. Thus, the inputs for this

problem are a set of student marks. The goal of this problem iss too print tthe pass and fail counts

ss

of the students, which is also the output of this algorithm. The process of reading a students

he proc

proce

mark is done manually. Compare the student mark with the pass m mark, that is, 50. If the student

ma

re

hould be ad

mark is greater than or equal to 50, then pass count should added by one. Otherwise, the fail

yP

count should be added by one. This process is repeatedpeated forr aall the students.

ated fo

The procedure that is done manually can bee given as an algorithm. Therefore, informally,

sit

the algorithm for this problem can be given follows:

en as follows

follow

Step 1: Let counter = 1, number of students

dents = 6

tudents

er

student

iv

students

2.2: Compare the marks the sstudent with 50

arks of th

Un

2.3: If student ma

mark greater than or equal to 50

rk is grea

Then increment

crement the ppass count

ment th

Else increment

rement tthe fail count

2.4: Increment th counter

crement the

Step 3: Print

nt pass count and fail count

ss coun

Step 4: Exit

xit

Thus, algorithmic

gorith solving can be observed to be much similar to how we solve problems

manually.

Some of the important characteristics are listed as follows:

Input An algorithm can have zero or more inputs.

Output An algorithm should produce at least one or more outputs.

'eniteness An algorithm is characterized by deniteness. Its instructions should be clear and

unambiguous without any confusion. All operations should be well dened. For example, opera-

tions involving the division of zero or taking a square root of a negative number are unacceptable.

8niTueness An algorithm should be a well-dened and ordered procedure that consists of

a set of instructions in a specic order. The order of the instructions is important as a change

in the order of execution leads to a wrong result or uncertainty.

KhW

Introduction to Algorithms 7

(IIectiYeness An algorithm should be effective, which implies that it should be traceable

manually.

Finiteness An algorithm should have a nite number of steps and should terminate after

executing the nite set of instructions. Therefore, niteness is an important characteristic of

an algorithm.

Some of the additional characteristics that an algorithm is supposed to possess are the

following:

6iPpOicit\ Ease of implementation is another important characteristic of an algorithm.

*eneraOit\ An algorithm should be generic, independent of any y pprogramming language

or operating systems, and able to handle all ranges of inputs;; itt shoul

should not be written for a

ss

specic instance.

re

An algorithm that is denite and effective is also called a computational

co

com procedure. In

addition, an algorithm should be correct and efcient,ent that is, should work correctly for all

yP

m that ex

valid inputs and yield correct outputs. An algorithm xec

executes fast but gives a wrong result

is useless. Thus, efciency of an algorithm iss secondary ccompared to program correctness.

sit

ailable

able for a ggiven problem, then one has to select

However, if many correct solutions are available

on)

n) based on factors such as speed, memory usage,

the best solution (or the optimal solution)

er

and ease of implementation.

iv

W

Un

programs. Algorithms, like blueprints that are used

for constructing a house, help

e,, he solve a given problem logically. A program is an expres-

lp sol

sion of that idea, consisting

ting of a se

nsisting set of instructions written in any programming language.

d

algorithms can also be contrasted with software development. At the

or

l, software

who developp software,

software often for third-party customers, on a commercial basis. Software

xf

engineering domain that deals with large-scale software development. Project manage-

ngg is a dom

do

O

ment issues such as team management, extensive planning, cost estimation, and project

scheduling are the main focus of software engineering. On the other hand, algorithm

design and analysis as a eld of study take a micro view of program development. Its

focus is to develop efcient algorithms for basic tasks that can serve as a building block

for various applications. Often project management issues of software development are

not relevant to algorithms.

Computer resources are limited. Hence, many problems that require a large amount of resources

cannot be solved. One good example is the travelling salesperson problem (TSP). Its brief

history is given in Box 1.3.

A TSP is illustrated in Fig. 1.2. It can be modelled as a graph. A graph consists of a set of

nodes and edges that interconnect the nodes. In this case, cities are the nodes and the paths

that connect the cities are the edges. Edges are undirected in this case. Alternatively, it is

KhW

8 Design and Analysis of Algorithms

A travelling salesperson problem (TSP) is one of the most holes called vertices. The objective is to nd a tour starting

important optimization problems studied in history. It is from a city, visiting all other cities only once, and nally

very popular among computer scientists. It was studied returning to the city where the tour has started. This is

and developed in the 19th century by Sir William Rowan called a Hamiltonian cycle. The TSP is to nd a Hamiltonian

Hamilton and Thomas Pennington Kirkman. In 1857, cycle in a graph. The problem was then studied extensively

Hamilton created an Icosian game, a pegboard with 20 by Karl Menger in Vienna and by Harvard later in 1920.

(a) (b) to any other city. The complete details

A B

of graphshs are pprovided in Chapter 5. A

TSP P involves

nvolves a travelling person who

ss

A B

starts

rts from a city, visits all other cities

re

only onc

once, and returns to the city from

wh

wher

where he started.

yP

C C D A brute force technique can be used

(c) (d) ffor solving this problem. Let us enu-

sit

Fig. 1.2 Travelling salesperson problem (a) One cityno yno

no path merate all the possible routes. For a

TSP involving only one city, there is

er

(b) Two cities (c) Three cities (d) Four cities

itiess

no path. For two cities, there is only

iv

one path (AB). For three cities, s, there are twotw paths. In Fig. 1.2, these two paths are ABC

and ACB, assuming that A is the o orig

origin from where the travelling salesperson started.

Un

re {ADBCA,

{AD

{A ADCBA, ABCDA, ABDCA,

ACDBA, and ACDB CDB A}

ACDBA}.

d

Thus, every addition ion of a city can be noted to increase the path exponentially. Table 1.2

ddition

or

er of poss

Therefore,e, it can beb observed that, for N cities, the number of routes would be (N 1)!

xf

availability of an algorithm for a problem does not mean that the problem is

O

solvable by y a computer.

com

co There are many problems that cannot be solved by a computer and

for many problems algorithms require a huge amount of resources that cannot be provided

practically. For example, a TSP cannot be solved in reality. Why? Let us assume that there

are 100 cities. As N = 100, the possible routes are then

(100 1)! = 99!.

Table 1.2 Complexity of TSP

The value of the number 50! is 3041409320171337804

Number of cities Number of routes 3612608166064768844377641568960512000000000000.

1 0 (as there is no route) Therefore, 99! is a very large number, and even if a computer

2 1 takes one second for exploring a route, the algorithm will

3 2

run for years, which is plainly not acceptable. Just imagine

how much time this algorithm will require, if the TSP is

4

tried out for all cities of India or USA. This shows that the

5 24

development of efcient algorithms is very important as

120

computer resources are often limited.

KhW

Introduction to Algorithms 9

Problem solving is both an art and a science. The problem-solving process starts with the

understanding of a given problem and ends with the programming code of the given problem.

The stages of problem solving are shown in Fig. 1.3.

The following are the stages of problem solving:

1. Understanding the problem 5. Analysing an algorithm

2. Planning an algorithm 6. Implementing an algorithm

3. Designing an algorithm 7. Performing empirical analysis (if necessary)

4. Validating and verifying an algorithm

1.5.1 Understanding the Prolem

Problem

The study of an algorithm starts with h the computability theory.

Theoretically, the primary question off compu computability theory is the

ss

following: Is the given problem solvable?olvable?

vable? TTherefore, a guideline for

re

problem solving is necessary to understan

understand the problem fully. Hence,

Planning

a proper problem statementt is rrequired.

equi

equire Any confusion regarding or

yP

misunderstanding of a problem sta statement will ultimately lead to a

wrong result. Often, solving

olving numerical instances of a given problem

ving the nu

num

sit

Algorithm design can give an insightt to problem. Similarly, algorithmic solutions of

o the pro

proble

a related problem em

m will pprov

provide more knowledge for solving a given

er

algorithm.

iv

Generally,y, compu

computer systems cannot solve a problem if it is not properly

dened.

ed. often fall under this category, which needs supreme level

d. Puzzles ofte

Un

off int

intelligence.

lligence. T

lligenc These problems illustrate the limitations of computing

Algorithm analysis

power.

ower. H Humans

uma are better than computers in solving these problems.

rd

1.5.2

o

The

xf

Implementationn and

alysis

empirical analysis of the important decisions are detailed in the following subsections.

O

D

A computation model or computational model is an abstraction of a

Post analysis

real-world computer. A model of computation is a theoretical math-

ematical model and does not exist in physical sense. It is thus a virtual

machine, and all programs can be executed on this virtual machine

Is theoretically. What is the need for a computing model? It is meaning-

OK less to talk about the speed of an algorithm as it varies from machine

No to machine. An algorithm may run faster in machine A compared to in

Yes machine B. This kind of an analysis is not correct as algorithm analysis

should be independent of machines. A computing model thus facilitates

Exit

such machine-independent analysis.

Fig. 1.3 Stages of First, all the valid operations of the model of computation should be

problem solving specied. These valid operations help specify the input, process, and

KhW

10 Design and Analysis of Algorithms

Alan Turing (19121954) is well known for his contribution a theoretical machine, called the Turing machine that is used

towards computing. He is considered by many as the father widely in computability theory and complexity analysis. Alan

of modern computing. His contribution towards computability Turing is also credited with the designing of the Turing Test

theory and articial intelligence is monumental. He designed as a measure of testing the intelligence of a machine.

output. In addition, the computing models provide a notion of the necessary steps to compute

time and space complexity of a given algorithm.

For algorithm study, a computation model such as a random access machine (RAM) or a

Turing machine is used to perform complexity analysis of algorithms. RAM is discussed in

Chapter 3 and Turing machines are discussed in Chapter 17. The contribution of Alan Turing

he con

cont

ls (refer

is vital for the study of computability and computing models refer to Box 1.4).

ss

O

re

Data structure concerns the way data and its relationships hips are sstored. Algorithms require data

yP

for solving a given problem. The nature of data and nd thei organization can have impacts on the

their org

efciency of algorithms. Therefore, algorithms ms and data sstructures together often constitute

sit

an important aspect of problem solving.

Data organization is something we are familiar with

re famili w in our daily life. Figure 1.4 shows

er

an example of data organization called ed a queu

qqueue. A gas station with one servicing point

should have a queue, as shown in n the gure,

gure to avoid chaos. A queue (or rst come rst

g

iv

ation wher the

th processing (lling of gas) is done in one end

Un

and a vehicle is added at the end. All vehicles have same priority in this case. Often

he other eend

a problem dictates thee choi

choicee of st

structures. However, this structure may not be valid in

cases of, for example,

ple,, handling

handl medical emergencies, where highest priority should be

d

or

anization is a very important issue that inuences the effectiveness of algorithms.

xf

At some point problem solving, careful consideration should be given to storing the data

oint of prob

pro

effectively. popular statement in computer science is algorithm + data structure = program.

y. A popu

O

ecti

ti of a data structure often proves fatal in the problem-solving process.

A wrong selection

Fuel station

Queue of trucks

KhW

Introduction to Algorithms 11

Algorithm design is the next stage of problem solving. The following are the two primary

issues of this stage:

1. How to design an algorithm? 2. How to express an algorithm?

Algorithm design is a way of developing algorithmic solutions using an appropriate design

strategy. Design strategies differ from problem to problem. Just imagine searching a name in a

telephone directory. If anyone starts searching from page 1, it is termed as a brute force strategy.

Since a telephone directory is indexed, it makes sense to open the book and use the index at the

top to locate the name. This is a better strategy. One may come across many design paradigms

in the algorithm study. Some of the important design paradigms are divide and conquer, and

dynamic programming. Divide and conquer, for example, divides thee pr pproblem into sub-problems

and combines the results of the sub-problems to get the nal solution lution oof the given problem.

ss

Dynamic programming visualizes the problem as a sequence decisions. Then it combines

ce of decis

the optimal solutions of the sub-problems to get an optimalmal global ssolution. Many design vari-

re

ants such as greedy approach, backtracking, and branch bound techniques have been dealt

ch and bou

yP

mportan

porta t iin developing efcient algorithms.

with in this book. The role of algorithm design is important

Thus, one important skill required for problem solving is th

sit the selection and application of suit-

able design paradigms. A skilled algorithm designer

signer is ca

called an algorist.

er

^

After the algorithm is designed, it should be communicated to a programmer so that the

iv

algorithm can be coded as a program. This sstage is called algorithm specication. Only three

ogram. Th

Un

unicating the idea of algorithms. One is to use natural languages

such as English to communicate

munic th aalgorithm. This is preferable. However, the natural lan-

unic te the

guage has some disadvantages

antages such as ambiguity and lack of precision. Hence, an algorithm

dvantages

d

is often written and communicated through pseudocode. Pseudocode is a mix of the natural

nd commu

commun

or

mathematics.

hematic Another way is to use a programming language. The problem with

xf

programming language notation is that readers often get bogged down by the programming

ng languag

code details.

ils. Therefore, the pseudocode approach is preferable. Even though there are no

s. There

O

f algorithm writing, certain guidelines can be followed to make algorithms

fo

more understandable. Some of these guidelines are presented in Chapter 2.

Algorithm validation and verication are the methods of checking algorithm correctness.

An algorithm is expected to give correct outputs for all valid inputs. Sometimes, algorithms

may not give correct outputs due to logical errors. Hence, validation of algorithms becomes

a necessity. Algorithm validation is a process of checking whether the given algorithm gives

correct outputs for valid inputs or not. This is done by comparing the output of the algorithm

with the expected output.

Once validation is over, algorithm verication or algorithm proving begins. Algorithm veri-

cation is a process of providing a mathematical proof that the given algorithm works correctly

for all instances of data. One way of doing it is by creating a set of assertions that are expressed

using mathematical logic. Assertions are statements that indicate the conditions of algorithm

KhW

12 Design and Analysis of Algorithms

variables at various points of the algorithm. Preconditions indicate the conditions or variables

before the execution of an algorithm, and postconditions indicate the status of the variables at

the end of the execution of an algorithm. Remember, still the algorithm has not been translated

or converted into a code of any programming language. Hence, all these executions are rst

carried out theoretically on a paper and proof for the algorithm is determined. A proof of an

algorithm is said to exist if the preconditions can be shown to imply the postconditions logically.

A complete proof is to write each statement clearly for proving that the algorithm is right. In

addition to assertions, mathematical proof techniques such as mathematical induction can be

used for proving that algorithms do work correctly. Mathematical induction is one such useful

proof technique that is often used to prove that the algorithm works for all possible instances.

Mathematical proofs are rigorous and always better than algorithm validation. Program cor-

rectness itself is a major study and extensive research is done in this

h eld.

ss

In algorithm study, complexity analysis is important as we aree mainly iinterested in nding optimal

re

algorithms that use fewer computer resources. Complexity exity theory

theor is a eld of algorithms that

the

yP

mputat

uta ional

deals with the analysis of a solution in terms of computational ion resources and their optimization.

Humans are more complex than, say, amoeba. So o what dodoes the word complexity refer to here?

Complexity is the degree of difculty associatedted

d with a problem

pro

prob and the algorithm. The complex-

sit

ity of an algorithm can be observed to be related

ated to iits iinput size. For example, an algorithm for

sorting an array of 10 numbers is easy, y, but the ssam

same algorithm becomes difcult for 1 million

er

etween cocomplexity and the size of the input.

comp

iv

fo the

th following two purposes:

Un

y of the giv

1. To decide the efciency given algorithms

hmss fo

2. To compare algorithms for decidi

dec

deciding the effective solutions for a given problem

d

ollowing scenario: for a problem =, two algorithms A and B exist. Find the

owing sce

or

optimal algorithm. m. To solve this, there must be some measures based on which comparisons can

hm.

be made. In general, two types of measures are available for comparing algorithmssubjective

neral, tw

xf

and objective.

ive. Subjective

ctive. Subje measures are factors such as ease of implementation, algorithm style,

O

and readability

ility oof the algorithm. However, the problem with these subjective measures is that

these factors cannot be quantied. In addition, a measure such as the ease of implementation,

style of the algorithm, or understandability of algorithms is a subjective measure that varies

from person to person. Therefore, in algorithm study, comparisons are limited to some objective

measures. Objective measures, unlike subjective measures, can be quantied. The advantages

of objective measures are that they are measurable and quantiable, and can be used for predic-

tions. Often time and space are used as objective measures for analysing algorithms.

Time complexity means the time taken by an algorithm to execute for different increasing

inputs (i.e., differently-scaled inputs). In algorithm study, two time factors are considered

execution time and run time. Execution time (or compile time) does not depend on problem

instances. Additionally, a program may be compiled many times. Therefore, time in an algo-

rithm context always refers to the run time as only this is characterized by instances. Another

complexity is space complexity, which is the measurement of memory space requirements

of an algorithm. Technically, an algorithm is efcient if lesser resources are used. Chapter 3

focuses on the analysis of algorithms.

KhW

Introduction to Algorithms 13

In algorithm study, time complexity is not measured in absolute terms. For example, one

cannot say the algorithm takes 3.67 seconds. It is wrong as time complexity is always de-

noted as a complexity function t(n) or T(N), where t(n) is a function of input size n and is

not an absolute value. The variables n and N are used interchangeably and always reserved

to represent the input size of the algorithm. Recollect that input size is the number of binary

bits used to represent the input instance. The logic here is that for larger inputs the algorithm

would take more time. For example, sorting a list of 10 elements is easy, but sorting a list

of 1 billion elements is difcult. Generally, algorithms whose time complexity function is

a polynomial, for example, say N or log N, can be solved easily, and but problems whose

algorithms have exponential functions, say 2N, are difcult to solve.

Example 1.1 Assume that there are two algorithms A and B for a given problem P. The

time complexity functions of algorithms A and B are, respectively,

ly,, 3n and 2n. Which algorithm

3n an

s

n the

should be selected assuming that all other conditions remain he same for both algorithms?

es

6oOution Assuming that all conditions remain the same for both algorithms, the best

Pr

algorithm takes less time when the input is changedged

d to a la

larger value. Results obtained

ble 1.3

employing different values of n are shown in Table 1.3.

y

sit

Table 1.3 Time complexities

plexities

exities of algorithms

alg A and B

er

Input size (n) hm

m A T(n)

Algorithm T(n) = 3n

3n T n) = 2n

T(

Algorithm B T(n)

1 3 2

iv

Un

5 15 32

10 30 1024

d

or

xf

It can bee observed that algorithm A performs better as n is increased; time complexity

nearly aand

increases linearly an gives lesser values for larger values of n. The second algorithm instead

O

enti

grows exponentially, and 2100 is a very large number.

Therefore, algorithm A is better than algorithm B.

Example 1.2 Let us assume that, for a problem P, two algorithms are possiblealgorithm

A and algorithm B. Let us also assume that the time complexities of algorithms A and B are,

respectively, 3n and 10n2 instructions. In other words, the instructions or steps of algorithms A

and B are 3n and 10n2 respectively. Here, n is the input size of the problem. Let the input size

n be 105 instructions. If the computer system executes 109 instructions per second, how much

time do algorithms A and B take?

6oOution Here, n = 105, and the computer can execute 109 instructions per second.

5

Therefore, algorithm A (time complexity 3n) would take 3 10 9

3

= 4 = 0.0003 seconds.

10 10

5 2

10 (10 ) 10 1010

Algorithm B (time complexity 10n2) would take = = 100 seconds.

109 109

KhW

14 Design and Analysis of Algorithms

It can be observed that algorithm A would take very less time compared to algorithm B.

Hence, it is natural that algorithm A is the preferred one.

One may wonder whether this has got anything to do efciency. Imagine that we are now

executing algorithm A on a slower machinea machine that executes only 105instructions.

What would be the scenario in this case?

5

Algorithm A would take 3 10 = 3 seconds.

105 5 2

We can see that algorithm A is still better compared to algorithm B that takes 10 (10

5

) =

10

106 seconds. Therefore, the important point to be noted here is that speed of the machine

does not affect selection of algorithm A as a better algorithm. Hence, while computer speed

is crucial, the role of a better designed algorithm is still signicant.

call Anal

Analy

Analysis

ss

After the algorithm is designed, it is expressed as a program and d a suitab

suitable programming language

re

is chosen to implement the algorithm as a code. After the he program is written, it must be tested.

Even if the algorithm is correct, sometimes the program ogram may not give expected results. This

yP

may be due to syntax errors in coding the algorithm thm in a programming language or hardware

ithm

fault. The error due to syntax problems of a languageanguage called a logical error. The process of

nguage is ca

sit

removing logical errors is called debugging. g. Iff the program

progra leads to an error even in the absence

pro

of syntax errors, then one has to go back ck to the de design stage to correct the algorithmic errors.

desig

er

Now, complexity analysis can be performperformed for the developed program. The analysis

gram

that is carried out after the program am is ddeve

developed is called empirical analysis (or a priori

iv

sis).

analysis or theoretical analysis). s). This ana

aanalysis is popular for larger programs; a new area,

Un

hmics,, whe

hmics

called experimental algorithmics, w

where analysis is performed for larger programs using a

dataset, is emerging. A dataset

datas

atas t is

i a huge collection of valid input data of an algorithm; the

hat iss often used for testing of algorithms is called a benchmark dataset.

standard dataset that

d

result

results of a program on a benchmark dataset provides vital statistical

or

information about ut the bbehaviour of the algorithm. This kind of analysis is called empirical

xf

analysis (also

also called a posteriori analysis). In addition, the term proling is often used to

denote thee proce

proces

process of running a program on a dataset and measuring the time/space require-

O

program empirically.

A problem-solving process ends with postmortem analysis. Any analysis should end with a valu-

able insight. The following are the questions that are often raised in this problem-solving process:

1. Is the problem solvable?

2. Are there any limits for this algorithm? Is there any theoretical limit for the efciency of

this problem?

3. Is the algorithm efcient, and are there any other algorithms that are faster than the current

algorithm?

Answers to these questions give a lot of insight into the algorithms that are necessary for

rening the design of algorithms to make them more efcient. The best possible or optimal

solution that is theoretically possible for a given problem species a lower bound of a given

problem. The worst-case estimate of resources required by the algorithm is called an upper

KhW

Introduction to Algorithms 15

bound. Theoretically, the best solution of a problem should be closer to its lower bound. The

difference between the lower and upper bounds should be minimal for a better solution. The

difference between the upper and the lower bounds is called an algorithmic gap. Technically,

this should be zero. However, in practice, there may be vast difference between an upper and

a lower bound. A problem-solving process tries to reduce this difference by focusing on better

planning, design, and implementation on a continuous basis.

As algorithms are important for us, let us classify them for further elaboration. No single

criterion exists for classifying algorithms. Figure 1.5 shows some of the criteria used for

classication of algorithms.

ss

Algorithms

re

yP

Classication Classication Classication

Cla Classication

based on based on sit based on area of based on

implementation design specialization

spec tractability

ssicatio oof algorithms

er

iv

on

Un

ne can cl

classi

classify the algorithms as follows

ZE/

E

E

d

scientic

scienti principle for problem solving. Take a problem, reduce it,

or

eduction

duction pr

and repeat the reduction pro

process till the problem is reduced to a level where it can be solved

ng recursion

directly. Using recursion, the problem is reduced to another problem with a decrease in input

xf

instance. The

he transfo

transformed problem is the same as the original one but with a different input,

O

ss than that of the original problem. The problem reduction process is continued

which is less

till the given problem is reduced to a smaller problem that can be solved directly. Then the

results of the sub-problems are combined to get the result of the given problem. This strategy

of problem solving is called recursion.

Recursive algorithms use recursive functions for creating repetitions required for solving

a given problem.

A good example of recursion is computing the factorial of a number n. Factorial of a number

can be computed using the recursive function as follows:

0 if n = 0

n! =

n (n 1)! for n 1

Thus, a recursive function has a base case and an inductive case. A base case is the simplest

problem that can be solved directly. In this factorial example, the base case is nding 0!, as 0!

can be solved directly. An inductive case is a recursive denition of the problem that captures

the essence of a problem reduction process. It can be observed from Fig. 1.6, which shows the

computation of 4!, that recursion works by the principle of work postponement or delaying the

KhW

16 Design and Analysis of Algorithms

4! rial function calls itself by reducing its input by

Reduce 1 till the program reaches 0! This is the base

3! case and the algorithm stops here; results of the

4321

Reduce

sub-problems are collected for computing the

2! nal answer of the given problem.

321

Reduce The correctness of recursive algorithms is

1! due to its close relationship with the concept

21 0! of mathematical induction. In other words, re-

cursion is a mirror of mathematical induction.

1 Non-recursive (or iterative) algorithms, on

Fig. 1.6 Computation of 4! using recursion the other hand, aree deductive in nature. Non-

recursive algorithmsms do not use the recursion

ithms

ss

concept but, instead, rely on looping constructs, such as for or while statement, to create repetition

hile statem

of tasks. Non-recursive (or iterative) algorithms and theirr ana lyses aare discussed in Chapter 2,

analyses

re

and recursive algorithms and their analyses in Chapter 3.

yP

EEW

An algorithm that is designed for a single processor

rocessor

cessor is cca

called a sequential algorithm.

sit

ystems

tems that

A parallel algorithm is designed for systems tha use

u a set of processors. The concept of

ocessing

essing are

parallel processing and distributed processing a interrelated. Parallel systems have mul-

er

osely. Distr

tiple processors that are located closely. Distribu

Distributed systems, on the contrary, have multiple

ifferent

ferent pl

processors that are located at different place

places separated by a vast distance geographically.

iv

loosely coupled systems, while parallel systems are

Un

ms. Distribu

called tightly coupled systems. Distri

Distributed algorithms are implemented in a distributed system

environment. Parallel algorith

algorithms are discussed in Chapter 20.

orithms ar

d

or

An exact algorithm

orithm

thm nds

nd the exact solutions for a given problem. Some problems are so

complex that

at nding ttheir exact solutions is difcult. Approximation algorithms (discussed

hat

xf

in Chapter

er 19 of this

th book) nd equivalent or approximate solutions for a given problem.

O

E

Deterministic algorithms always provide xed predictable results for a given input. In con-

trast, non-deterministic algorithms or randomized algorithms take a different approach. For

deterministic algorithms, the output should always be true. On the other hand, randomized

algorithms relax this condition. It is argued that outputs based on random decisions may not

often result in correct answers or the algorithm may not terminate at all. Thus, the accuracy

of an output is associated with a probability. In daily life, we often use random decisions,

for example, in games such as dice. Industries conduct random quality checks on products.

Randomized samples are used to predict poll results and so on. One can be very sceptical

about randomized algorithms. However, randomized algorithms have been proved to be very

effective. Randomized algorithms are discussed in Chapter 19.

Based on design techniques, algorithms can be classied into various categories. As discussed

earlier, every design technique uses a strategy for solving a problem. Algorithms can be

KhW

Introduction to Algorithms 17

classied based on design as brute force, divide and conquer, dynamic programming, greedy

approach, backtracking, and branch and bound algorithms. This textbook is organized based

on this classication of algorithms only.

Algorithms can be classied based on the area of specialization. General algorithms such as

searching and sorting, and order statistics such as nding mean, median, rank, and so on are

useful for all elds of specialization. These algorithms can be considered as building blocks

of any area of specialization. Other than these basic algorithms, some algorithms are special-

ized for a particular domain. String algorithms, graph algorithms, combinatorial algorithms,

and geometric problems are some examples of specialized algorithms that are discussed in

this book. Let us discuss some of these algorithms now.

'

ss

Sorting algorithm is a general algorithm that is useful in all areas oof specialization. Sorting

re

problem is a structural problem. It involves structuring or rearrang

rearranging the sequence in a spe-

cic order. The importance of this problem arises fromom fact that all modern applications

m the fac

yP

require sorting. For example, an organization may y require

equi so ssorting of employee records based

on employee identication number (called primary

sitmary key). A library system may want books to

be ordered based on titles or ISBN numbers. Sorting

orting prob

pr

problems for small instances are relatively

easy. When sorting is required for a larger number, ssay a billion elements, designing sorting

gerr number

er

algorithms becomes more challenging. ng.

iv

^

Let us discuss some of these domain-spec

domain-sp

domain-specic algorithms now:

Un

g al orith are used frequently in document editing, web searches,

orithms

and pattern matching. g.

A sequence off character

characters is called a string. The sequence can be text, bits, numbers, or

gene sequences es (A,

A, C, G, or T). One problem that is of considerable interest is string match-

ing, which takes a pattern,

patt

pat searches the text string, and reports whether a pattern is present

or not. String

ring algorithms,

ng algo

algor which are discussed in Chapter 16, constitute a very important

eld of study.

y

Graph algorithms A graph is used for modelling complex systems by representing the rela-

tionships among the subsystems. A graph represents a set of nodes and edge. The nodes are also

called vertices. The nodes are connected by edges, which are also known as arcs. Consider the

graph shown in Fig. 1.7. Here, the nodes are given as {A, B, C, D, E} and the edges as {(A,B),

(A,C), (A,D), (B,A), (B,E), (B,C), (C,A), (C,B), (C,D), (C,E), (D,A), (D,C), (D,E), (E,B), (E,C),

and (E,D)}.

A

Many interesting problems can be formulated using this graph structure. The

following are some of the interesting graph algorithms:

B C D

7raYeOOing saOesperson proEOeP As stated earlier, a salesperson starts from a

node, visits all other nodes only once, and gets back to the starting node. This

E

problem is reduced to a Hamiltonian cycle if the graph is undirected. In the

case of a weighted graph (where edges carry some weights), the TSP is about

Fig. 1.7 Sample graph nding a tour of minimum cost.

KhW

18 Design and Analysis of Algorithms

Chinese postman problem This problem is the same as a TSP, but instead of vertices, an

edge should be visited only once. There is no restriction on vertices in this problem.

Graph colouring problem This problem is about how to colour all the vertices distinctly using

only a small number of colours such that no two neighbouring vertices share the same colour.

Combinatorial algorithms The focus of combinatorial algorithms is to nd a combinatorial

object inherent in the problem such as permutations, combinations, or a subset that satises

some constraints and objective functions. These problems are difcult to solve, and many

problems do not have algorithmic solutions. TSPs and graph colouring problems are examples

of combinatorial problems.

Geometric algorithms Geometric algorithms deal with geometrical objects such as points,

lines, and polygons. The following are some of the algorithms discussed in this textbook:

Closest pair problem This problem deals with nding the distance tance bbetween points in a 2D

ss

space and nding a pair of points that are closest to each other

other.

her..

re

Convex hull problem This problem deals with nding smallest convex polygon that

ding the sm

includes all points in a 2D space.

yP

1.6.4 Based on Tractability sit

Tractability means solvability of a given n problem wiwithin a reasonable amount of time and

cultt to solv

space. An intractable problem is difcult solve w

within the reasonable amount of computer

er

hee following

resources. Based on tractability, the followin categories are possible:

iv

problem

problems have polynomial time complexity or their upper

Un

polynomial. These problems are solvable. For example, sorting

polynom

is an example of solvable polynomial problems.

blee or polynom

polyn

ems Prob

Unsolvable problems Problems such as halting problems cannot be solved at all. These

d

vable or no

are called unsolvable non

non-computable problems. Knowledge about the non-computability

or

lems

ms helps in project management.

of these problems

xf

ble

Intractable problem These problems are of two categories. One category comprises a

le problems

blems

lems that

set of problems tth have algorithmic solutions but require more computer resources. Hence,

O

these solutions are practically non-implementable. In addition, these have been proved to be

computationally hard. The other category consists of a set of problems that have been proved

to be computationally hard. For example, a TSP is a proven computationally hard problem;

it had already been discussed that time complexity of the algorithm increases exponentially

with an increase in the number of cities.

KhW

Introduction to Algorithms 19

Thus, algorithm study can be concluded as the central theme of computer science. An as-

piring computer professional should be a better problem solver. The problem-solving process

starts with the understanding of a given problem and ends with nding an efcient program-

ming code for the given problem. Problem solving poses many challenges as it is a creative

process. Chapter 2 introduces the basics of a problem-solving process and also introduces

basic guidelines for writing algorithms.

Q SUMMARY Q

An algorithm is a step-by-step procedure for solving a Algorithm verication is a process of providing a math-

given problem. ematical proof that the given algorithm works correctly

A computational problem is characterized by two for all instances of data.

factorsspecication of valid input and output param- A proof of an algorithm m is said to exist if the precondi-

eters of the algorithm, and specication of the relation- tions can be shown to implymply postconditions

po logically.

ship between inputs and outputs. An estimation off thee time andaan space complexities of

An algorithm that yields the correct output for a legal an algorithm for varying input

in sizes is called algorithm

input is called an algorithmic solution. analysis.

The art of designing, analysing, and implementing Time complexity

omplex

le ty re

ref

refers to the measurement of run time

algorithms is called algorithmics. of an algorithm in terms of its input size, and space

Algorithms can be contrasted with programs. A program mplexity is the

complexity tth measurement of space required for

is an expression of algorithm in a programming language. a given a algorithm.

algo

A valid input is called an instance. The number of binary A data

datase

dataset is a huge collection of valid input data

bits necessary to encode inputs of an algorithm is called ed of an

a algorithm. The standard dataset that is often

the input size. used for testing of algorithms is called a benchmark

use

An algorithm should have characteristics such uch as a dataset.

well-dened order, inputs, outputs, niteness, ess, denite-

ss, denite The theoretically best possible or optimal solution for

ness, efciency, and generality. a given problem species a lower bound of a given

Problem solving starts with the understanding

nderstan

erstan iing of

o a problem. The worst-case estimate of resources that can

problem statement without any confusion. fusion. be required by an algorithm is called an upper bound.

A computation or computational nal model is an abstraction The difference between the upper and the lower bound

of a real-world computer. is called an algorithmic gap.

Algorithm design is a wayy of developing

deve

dev algorithmic Problem reduction is a scientic principle for problem

solutions using a suitable

itable course

uitable cour of action called a

cours solving. Take a problem, reduce it, and repeat the re-

design strategy. Algorithm

rithm specication

sp is about com- duction process till the problem is reduced to a level

municating the design strategy

tra to a programmer often where it can be solved directly.

in the form of a pseudocode. Algorithms can be categorized based on their imple-

Algorithmic validation means checking whether the mentation methods, design techniques, eld of study,

algorithm gives a correct result or not. and tractability.

Q GLOSSARY Q

Agent A performer of an algorithm $OJRULWKPYHULFDWLRQ The process of providing a mathemati-

Algorist A person who is skilled in algorithm development cal proof that the algorithm works correctly for all valid inputs

Algorithm A step-by-step procedure for solving a given Algorithm validation The process of checking the correct-

problem ness of an algorithm, this is done by giving valid inputs to

Algorithm gap The difference between lower and upper it and checking its results with expected values

bounds Approximation algorithms Algorithms that provide ap-

$OJRULWKPVSHFLFDWLRQ Formalization of an algorithm in proximate solutions for problems whose exact solutions

a suitable form that can be conveyed to a programmer are difcult to obtain

KhW

20 Design and Analysis of Algorithms

Complexity theory $ HOG RI DOJRULWKP VWXG\ WKDW GHDOV Non-recursive algorithm (iterative algorithm) $Q DOJRULWKP

ZLWK WKH DQDO\VLV RI DOJRULWKPV WKDW LV GHYHORSHG XVLQJ LWHUDWLYH FRQVWUXFWV IRU FUHDWLQJ

Computability theory $ HOG RI VWXG\ WKDW GHDOV ZLWK WKH UHSHWLWLRQ RI WDVNV LQVWHDG RI XVLQJ UHFXUVLYH IXQFWLRQV

WKHRU\ RI DOJRULWKPV DQG VROYDELOLW\ RI WKH DOJRULWKPV Parallel algorithm $Q DOJRULWKP GHVLJQHG IRU V\VWHPV WKDW

Computational model $Q DEVWUDFWLRQ RI D UHDO-ZRUOG FRP- XVH PXOWLSOH SURFHVVRUV

SXWHU V\VWHP 3UROLQJ 7KH SURFHVV RI PHDVXULQJ WLPH DQG VSDFH FRP-

Computational problem $ SUREOHP WKDW FDQ EH VROYHG E\ SOH[LW\ E\ UXQQLQJ D SURJUDP RQ D ODUJHU GDWDVHW

FRPSXWHU V\VWHPV Proof 7KH ORJLFDO GHULYDWLRQ VKRZLQJ WKDW WKH SUHFRQGLWLRQV

Deterministic algorithm $OJRULWKPV WKDW DOZD\V JLYH WKH RI DOJRULWKPV LPSO\ WKHLU SRVWFRQGLWLRQV

VDPH UHVXOW IRU D [HG LQSXW Recursive algorithm $Q DOJRULWKP WKDW XVHV D UHFXUVLYH

Exact algorithm $Q DOJRULWKP WKDW LV GHVLJQHG WR JLYH DQ IXQFWLRQ

H[DFW VROXWLRQ IRU D JLYHQ SUREOHP Sequential algorithm $Q DOJRULWKP WKDW LV GHVLJQHG IRU D

Experimental algorithmics $ HOG RI VWXG\ WKDW GHDOV ZLWK VLQJOH-SURFHVVRU V\VWHP

WKH DQDO\VLV RI DOJRULWKPV H[SHULPHQWDOO\ XVLQJ D ODUJH Space complexity 0HDVXUHPHQW RI VSDFH UHTXLUHG IRU

GDWDVHW DQ DOJRULWKP

Intractable problems 3UREOHPV WKDW FDQQRW EH VROYHG ZLWKLQ Strategy $ ZD\ RI GHVLJQLQJLJQLQJ DOJRULWKPV

D

ss

D UHDVRQDEOH DPRXQW RI FRPSXWHU UHVRXUFHV Time complexity 5XQ WLPH PHDVXUHPHQW RI WKH DOJRULWKP

Lower bound $ WKHRUHWLFDO EHVW VROXWLRQ IRU D JLYHQ SUREOHP ZKHQ WKH LQSXWS LV VFDOHG

SXW VFDOH WR D ODUJHU YDOXH

re

Non-deterministic algorithm $Q DOJRULWKPV WKDW XVHV UDQ- oblems 3UREOHPV

Tractable problems 3 IRU ZKLFK DOJRULWKPV WKDW

yP

GRPL]HG FKRLFHV WKDW GHWHUPLQH WKH FRXUVH RI H[HFXWLRQ QG VROXWLRQV

WLR V ZLWKLQ

ROXWLR Z

ZLWK D UHDVRQDEOH DPRXQW RI FRPSXWHU

RI LQVWUXFWLRQV KHQFH WKH RXWSXW RI DQ DOJRULWKP YDULHV VRXUFHV

RXUFHV H[LVW

UHVRXUFHV H

H[L

HYHQ IRU D [HG LQSXW pper bound 7

er boun

Upper 7KH ZRUVW-FDVH FRPSOH[LW\ RI DQ DOJRULWKP

sit

REVIEW QUESTIONS Q

W QUESTION

QUEST

er

Q

:KDWLVDQDOJRULWKPVSHFLFDWLRQ":KDWLVDSVHXGR-

iv

Un

YH DOJRULWKPV

DOJRULWK

6XUYH\WKH,QWHUQHWDQGOLVWRXWDWOHDVWYHDOJRULWKPV :KDWLVWKHGLIIHUHQFHEHWZHHQDOJRULWKPYHULFDWLRQ

LYHV

YHV

WKDWKDYHKXJHLPSDFWRQRXUGDLO\OLYHV DQGDOJRULWKPYDOLGDWLRQ"

VROYLQJ"

YLQJ"

:KDWDUHWKHVWDJHVRISUREOHPVROYLQJ" +RZLVDQDOJRULWKPYDOLGDWHGDQGYHULHG"([SODLQ

d

:KDWLVPHDQWE\WLPHDQGVSDFHFRPSOH[LWLHV"

SDFH FRPSOH[ ZLWKDQH[DPSOH

or

DUDGLJP.

UDGLJP.

'HQHWKHWHUPGHVLJQSDUDGLJP :KDWDUHWKHFULWHULDXVHGIRUFODVVLFDWLRQRIDOJRULWKPV"

xf

Q EXERCISES Q

O

$VVXPHWKDWWKHUHDUHWZRDOJRULWKPV$DQG%IRUD

WZ FRPSOH[LWLHV RI$ % DQG & DUH n n DQG ORJ n

JLYHQSUREOHP37KHWLPHFRPSOH[LW\IXQFWLRQVRIDOJR- UHVSHFWLYHO\$VVXPHWKDWWKHLQSXWLQVWDQFHnLV3.

ULWKPV$DQG%DUHUHVSHFWLYHO\nDQGORJ2n:KLFK $VVXPH WKDW WKH PDFKLQH H[HFXWHV 9 LQVWUXFWLRQV

DOJRULWKPVKRXOGEHVHOHFWHGDVVXPLQJWKDWDOORWKHU SHU VHFRQG. +RZ PXFK WLPH ZLOO DOJRULWKPV $ % DQG

FRQGLWLRQVUHPDLQWKHVDPHIRUERWKWKHSUREOHPV" & WDNH" :KLFK DOJRULWKP ZLOO EH WKH EHVW"

2 /HWXVDVVXPHWKDWIRUDWHOHSKRQHGLUHFWRU\VHDUFK

SUREOHP3WKUHHDOJRULWKPVH[LVW$%DQG&7LPH

Q ADDITIONAL PROBLEM Q

. -RKQ 0DF&RUPLFN KDG ZULWWHQ D ERRN WLWOHG Nine UHFRJQLWLRQ GDWD FRPSUHVVLRQ GDWDEDVHV DQG GLJLWDO

Algorithms That Changed the Future: The Ingenious VLJQDWXUHV WKDW FKDQJHG WKH ZRUOG.

Ideas that drive Todays Computers, 3ULQFHWRQ 8QLYHUVLW\ D :KDW DUH WKHVH DOJRULWKPV" 6HDUFK WKH ,QWHUQHW

3UHVV 3ULQFHWRQ WKDW KDG OLVWHG WKH QLQH ZRQGHUIXO DQG QG ZKDW WKHVH DOJRULWKPV DUH IRU.

DOJRULWKPV QDPHO\ VHDUFK HQJLQH LQGH[LQJ SDJH UDQN E ,GHQWLI\ RQH PRUH DOJRULWKP WKDW \RX IHHO FKDQJHV

SXEOLF NH\ FU\SWRJUDSK\ HUURU FRUUHFWLQJ FRGHV SDWWHUQ WKH ZRUOG ZH OLYH LQ.

KhW

Introduction to Algorithms 21

Q CROSSWORD Q

1

2

3

4 5

7

8

9 10

s

11

es

y Pr

12 sit

13

er

iv

Across Down

D

Dow

ble

1. Algorithms that are solved within reasonable e amount 1.

1 Measurement of time

Un

ons

4. Algorithms that use recursive functions 3. A step-by-step procedure

. Who coined the word algorithm analysisysis 5. A person skilled in algorithms

d

or

ing resources

suring resourc on a larger

resou

dataset

xf

11. Theory of algorithms

O

13. Attributed as Father of Modern Computing by many

KhW

- Exercises of Design & AnalysisUploaded byAndyTrinh
- Optimal-Storage-on-TapesUploaded byapi-19981779
- MQL4 ProgrammingUploaded byKhairuddin Ismail
- november2002paper2Uploaded byapi-3753397
- (2) How Should I Practice So That I Will Be at a Level Where I Can Approach TopCoder's Div1-500 Problems With ConfidenceUploaded byKrutarth Patel
- (6) What is the Maths Require for the Competitive Programming_ - QuoraUploaded byRajib Bose
- 07 Good Programming StyleUploaded byShreya Sagar
- Implementing Quicksort ProgramsUploaded byravg10
- Tmp 3078Uploaded byFrontiers
- Scimakelatex.95793.Md.+Sharafat+JamilUploaded bySharafat Jamil
- GATE SyllabusUploaded byAnkit Jain
- pajekUploaded byDiego Astorga
- Designing Good Algorithms for MapReduce and BeyondUploaded bylcm3766l
- Scimakelatex.15413.Anon.mousUploaded bymdp anon
- Lecture 3Uploaded byMuhammad Dureahmad
- Meidan Filter DocUploaded bytrungkiendt9
- Es 15 LibroUploaded byasasasasas1
- Electronic HomeworkUploaded byAmalia Goia
- Analyzing B-Trees and Online AlgorithmsUploaded byAuthor
- The Effect of Autonomous Information on Electrical EngineeringUploaded byJames
- TS PGECET CSE and IT syllabus & Scheme of ExaminationUploaded bypavani
- AP PGECET CSE-IT Exam Syllabus and PatternUploaded bypavani
- OD6 PL Dynamic Programming BUploaded bycarolinarvsocn
- AlgorithmUploaded bypriya
- Development of Operating SystemsUploaded byTimothy Parks
- Unit 1 -lastUploaded bynsavi16edu
- Review an Analytical Solution to Fuzzy TOPSIS and Its Application in PersonnelUploaded byBagas Laksono
- Scimakelatex.10021.NoneUploaded byJohn Miller
- NoticeUploaded byPritam Bhattacharya
- Searching And Sorting ProgramsUploaded bygurusodhii

- FastICAUploaded bySomdeb Seal
- The Set-covering ProblemUploaded bymouhabalde
- Low-Power Split-Radix FFT Processors Using Radix-2 Butterfly UnitsUploaded byjameelahmad
- MDSP1Uploaded byLatha Venkatesh
- Neural%20Networks_1.pdfUploaded byHassan Ali
- Chaeat SheetUploaded bySheikh Asher
- 4 Look Last LayerUploaded byPCwizCube
- Colored Gaussian Noise 2Uploaded byYasser Naguib
- Lecture 3 - Linear Methods for ClassificationUploaded bymissinu
- Effects Zero Pole on system responseUploaded byfancy_mask
- System Identification MatlabUploaded bykarlg
- Channel Equalization using Wiener Filter.pdfUploaded byanon_113257455
- Chapter 4 Digital Modulation_Part 2_2_0Uploaded bykimberly
- The Matlab Ode SuiteUploaded byflausen
- 2210b3fbc1f4b8246437a88a668bf9a0d2c0Uploaded byAnkita Saha
- DSP Digital Signal Processing MODULE I PART2Uploaded byARAVIND
- ON room correction (audio)Uploaded byAnthony Hubbard
- Algorithms Project ReportUploaded byapi-3722999
- Penyelesaian Menggunakan Matlab FandyUploaded byNizamMuluk
- chapindxf9Uploaded byJuan Perez
- pptUploaded bynarendranvel
- L1Uploaded byscualoboy2006
- Low Complexity Implementation of LDPC Decoder UsingUploaded bysamsath10
- Error Detection and CorrectionUploaded byaubaidullah muzammil
- An Effivient Iris Recognition Mathod Based on Gabor Wavelet Neural Network2Uploaded byVipin Kumar
- Geeks for Geeks -ArrayUploaded byLove Mehta
- FIRST COME FIRST SERVEUploaded byKAVIARASU.D
- Channel Coding TheoremUploaded byHossamSalah
- Digital Signal Processing NotesUploaded byDanial Zaman
- Signals and systemsUploaded byBikshu11

## Much more than documents.

Discover everything Scribd has to offer, including books and audiobooks from major publishers.

Cancel anytime.