Subject – Design and Analysis of Algorithms (BTCS-403-18)

Date – 13/4/2020
Unit Name – Advanced Topics
Unit no. 5
No. of Lessons in Unit – 3
Lessons Discussed – 1, 2, 3

Instructions to the students – Go through the notes and Ask queries (if any)
on my number 9896439120 (PM)

Name of Topics covered – Approximation Algorithms, Randomized

Algorithms, Heuristics and their characteristics.

Contents – MCQs, Answer Keys of Previous Lesson MCQs, Notes, Short

Answer Questions, Video Lecture Links, References

13th April MCQs Answer key-

1. B
2. A
3. C
4. C
5. A
6. A
7. C
8. B

Approximation Algorithms, Randomized Algorithms and Heuristics –

Deterministic vs. Randomized

One important (and exclusive) distinction one can make is, whether the algorithm is
deterministic or randomized.

Deterministic algorithms always produce the same results on a given input following the
same computation steps.

Randomized algorithms throw coins during execution. They are based on randomization in
input, sequence of instructions and even in case of output. Hence either the order of
execution or the result of the algorithm might be different for each run on the same input.
There are subclasses for randomized algorithms. Monte Carlo type algorithms and Las
Vegas type algorithms.

A Las Vegas algorithm will always produce the same result on a given input. Randomization
will only affect the order of the internal executions.

In the case of Monte Carlo algorithms, the result might change, even be wrong. However, a
Monte Carlo algorithm will produce the correct result with a certain probability.

So of course the question arises: What are randomized algorithms good for? The
computation might change depending on coin throws. Monte Carlo algorithms do not even
have to produce the correct result.

Why would that be desirable?

The answer is twofold:

• Randomized algorithms usually have the effect of perturbing the input. Or put it
differently, the input looks random, which makes bad cases very seldom.

• Randomized algorithms are often conceptually very easy to implement. At the same
time they are in run time often superior to their deterministic counterparts.

Usually algorithms have an optimization goal in mind, e.g. compute the shortest path or the
alignment or minimal edit distance.

Exact algorithms aim at computing the optimal solution given such a goal. Often this is quite
expensive in terms of run time or memory and hence not possible for large input.

In such cases one tries other strategies.

Approximation algorithms aim at computing a solution which is for example only a certain,
guaranteed factor worse than the optimal solution, i.e. an algorithm yields a c − approximation,
if it can guarantee that its solution is never worse than a factor c compared to the optimal
solution.

Alternatively, heuristic algorithms try to reach the optimal solution without giving a guarantee
that they always do. Often it is easy to construct a counter example.

A good heuristics is almost always near or at the optimal value.

 
A fresh start … 
There  are  two  approaches  to  getting  around  NP‐completeness.  First,  if  the  actual 
inputs  are  small,  an  algorithm  with  exponential  running  time  may  be  perfectly 
satisfactory.  Second,  it  may  still  be  possible  to  find near‐optimal solutions  in 
polynomial  time  (either  in  the  worst  case  or  on  the  average).  In  practice,  near‐
optimality is often good enough. An algorithm that returns near‐optimal solutions is 
called  an approximation  algorithm.  This  chapter  presents  polynomial‐time 
approximation algorithms for several NP‐complete problems. 
 
An algorithm that uses random numbers to decide what to do next anywhere in its 
logic  is  called  Randomized  Algorithm.  For  example,  in  Randomized  Quick  Sort,  we 
use  random  number  to  pick  the  next  pivot  (or  we  randomly  shuffle  the  array). 
Typically, this randomness is used to reduce time complexity or space complexity in 
other standard algorithms. 
 
1. Deterministic algorithm is an algorithm in which for given particular input it will
always produce the same output, with the underlying machine always passing
through the same sequence of states as shown in figure 1.

1. To overcome the computation problem of exploitation of running time of a

deterministic algorithm, randomized algorithm is used.
2. Randomized algorithm uses uniform random bits also called as pseudo random
number as an input to guides its behavior (Output).
3. Randomized algorithms rely on the statistical properties of random numbers. One
example of a randomized algorithm is quicksort.
4. It tries to achieve good performance in the average case.
5. Randomized algorithm is also called as probabilistic algorithm.
6. Randomized algorithm flow steps is shown figure 2.
 
Advantage of Randomized Algorithm:

1. The algorithm is usually simple and easy to implement.

2. The algorithm is fast with very high probability, and/or it produces optimum output
with very high probability.

A heuristic is a mental short cut or "rule of thumb" that gives some guidance on
how to do a task, but it does not guarantee solutions consistently. They often
reflect some form if inductive reasoning.
 
A heuristic algorithm is one that is designed to solve a problem in a faster and more
efficient fashion than traditional methods by sacrificing optimality, accuracy, precision,
or completeness for speed. Heuristic algorithms often times used to solve NP-complete
problems, a class of decision problems. In these problems, there is no known efficient
way to find a solution quickly and accurately although solutions can be verified when
given. Heuristics can produce a solution individually or be used to provide a good
baseline and are supplemented with optimization algorithms. Heuristic algorithms are
most often employed when approximate solutions are sufficient and exact solutions are
necessarily computationally expensive. 
 
Examples of Heuristics Algorithms
The following are well-known examples of “intelligent” algorithms that use clever simplifications and methods to
solve computationally complex problems.

Swarm Intelligence

Swarm Intelligence systems employ large numbers of agents interacting locally with one another and the
environment. Swarm intelligence refers to the collective behavior of decentralized systems and can be used to
describe both natural and artificial systems. Specific algorithms for this class of system include the particle
swarm optimization algorithm, the ant colony optimization algorithm, and artificial bee colony algorithm. Each of
the previous algorithms was inspired by the natural, self-organized behavior of animals.1

Tabu Search

This heuristic technique uses dynamically generated tabus to guide the solution search to optimum solutions. It
examines potential solutions to a problem and checks immediate local neighbors to find an improved solution.
The search creates a set of rules dynamically and prevents the system from searching around the same area
redundantly by marking rule violating solutions as “tabu” or forbidden. This method solves the problem of local
search methods when the search is stuck in suboptimal regions or in areas when there are multiple equally fit
solutions.2

Simulated Annealing

Borrowing the metallurgical term, this technique converges to a solution in the same way metals are brought to
minimum energy configurations by increasing grain size. Simulated annealing is used in global optimization and
can give a reasonable approximation of a global optimum for a function with a large search space. At each
iteration, it probabilistically decides between staying at its current state or moving to another while ultimately
leading the system to the lowest energy state.2

Genetic Algorithms

Genetic algorithms are a subset of a larger class of evolutionary algorithms that describe a set of techniques
inspired by natural selection such as inheritance, mutation, and crossover. Genetic algorithms require both a
genetic representation of the solution domain and a fitness function to evaluate the solution domain. The
technique generates a population of candidate solutions and uses the fitness function to select the optimal
solution by iterating with each generation. The algorithm terminates when the satisfactory fitness level has
been reached for the population or the maximum generations have been reached.2

Artificial Neural Networks

Artificial Neural Networks (ANNs) are models capable of pattern recognition and machine learning, in which a
system analyzes a set of training data and is then able to categorize new examples and data. ANNs are
influenced by animals’ central nervous systems and brains, and are used to solve a wide variety of problems
including speech recognition and computer vision.1

Support Vector Machines

Support Vector Machines (SVMs) are models with training data used by artificial intelligence to recognize
patterns and analyze data. These algorithms are used for regression analysis and classification purposes.
Using example data, the algorithm will sort new examples into groupings. These SVMs are involved with
machine learning, a subset of artificial intelligence where systems learn from data, and require training data
before being capable of analyzing new examples.
Short Answer Type Questions –  
1. Write a short note on the following:
i) Approximation Algorithms
ii) Reduction Techniques in NP Completeness
2. Explain Randomized Algorithm with special emphasis to their significance?
3. What are Heuristics? List and explain their characteristics?
4. Elaborate the terms satisfiability, reducibility and non-deterministic polynomial
and approximation in context of NP Completeness Theory.
5. What are reduction techniques? What is their significance?

Video Link
https://www.youtube.com/watch?v=JD5GzURbCdU 
https://www.youtube.com/watch?v=jFW7fwa0Zm8 

Reference / Useful Link-

http://staff.ustc.edu.cn/~csli/graduate/algorithms/book6/chap37.htm  
http://www.mi.fu‐berlin.de/wiki/pub/ABI/DiscretMathWS10/runtime.pdf 
https://universe.bits‐pilani.ac.in/uploads/A.Hyd%202014‐15/elgga/subruk_randomtalk.pdf 
http://cs.rkmvu.ac.in/~sghosh/public_html/nitrkl_igga/randomized‐lecture.pdf 
http://www‐personal.umich.edu/~mepelman/teaching/IP/Handouts/Handout8.pdf