IS239- Algorithms and Complexity (8 Credits)

Pre-Requisites: IS143 and IS237

Facilitator: Dr Godfrey Justo

 IS239: Course Outline
• Extended Data Structures & Algorithms Analysis Review (Refer IS237 + IS137)
• Maths for Algorithm Analysis
• Data structures: Binary tree, Hashing
• Sorting Algorithms: Insertion sort, selection sort, Quick-sort, merge-sort, heap-sort
• Comparison of sorting algorithms
• Application of Binary tree: Huffman coding
• Algorithmic Strategies
• Graph Algorithms: Graph definition, implementations, traversal(search), Connected components,
Topological sort, path problems, spanning trees, network flow, matching.
• Fundamental Algorithms: Greedy, divide and conquer, dynamic programming, randomized algorithms
• Distributed Algorithms: Introduction and applications, Distributed computations algorithms, merge
and sort; Complexity measures: time, space and message complexity
• Basic Computability
• Concepts in computability - computable and un-computable sets and functions
• Automata and Turing machine, Turing computability
• P versus NP: Classes of algorithms; NP-complete problems; NP-completeness and the classes P and
NP
 Introduction to Computability
• Computability is the ability to solve a problem in an effective manner
Stems from the field of computability theory within mathematical logic and
the theory of computation within computer science
The computability of a problem is closely linked to the existence of an algorithm to
solve the problem
Most widely studied models of computability which have computationally equivalent
power
 Turing-computable
 μ-recursive functions
Lambda calculus
Other forms of computability are studied as well:
Computability notions weaker than Turing machines are studied in automata theory
Computability notions stronger than Turing machines are studied in the field of hypercomputation
 Turing computability
• The main form of computability studied in recursion theory was introduced by
Turing (1936)
 A set of natural numbers is said to be a computable set (also called a decidable, recursive,
or Turing computable set) if there is a Turing machine that, given a number n, halts with
output 1 if n is in the set and halts with output 0 if n is not in the set
 A function f from the natural numbers to themselves is
a recursive or (Turing) computable function if there is a Turing machine that, on input n, halts
and returns output f(n)
• The use of Turing machines here is not necessary; there are many other models of
computation that have the same computing power as Turing machines
 Example the μ-recursive functions obtained from primitive recursion and the μ operator
 Automata and Turing machine
• Automata Theory is a branch of computer science that deals with designing
abstract self propelled computing devices that follow a predetermined
sequence of operations automatically
The term "Automata" is derived from the Greek word "αὐτόματα" which means "self-
acting“
 An automaton (Automata in plural) is an abstract self-propelled computing device
which follows a predetermined sequence of operations automatically
• An automaton with a finite number of states is called a Finite
Automaton (FA) or Finite State Machine (FSM)
 Formal definition of a Finite Automaton
• An automaton can be represented by a 5-tuple (Q, ∑, δ, q0, F), where −
Q is a finite set of states
∑ is a finite set of symbols, called the alphabet of the automaton
δ is the transition function
q0 is the initial state from where any input is processed (q0 ∈ Q)
F is a set of final state/states of Q (F ⊆ Q)
 Related Terminologies
• Alphabet
An alphabet is any finite set of symbols
Example : ∑ = {a, b, c, d} is an alphabet set where ‘a’, ‘b’, ‘c’, and ‘d’
are symbols
• String
A string is a finite sequence of symbols taken from ∑
Example: ‘cabcad’ is a valid string on the alphabet set ∑ = {a, b, c, d}
• Length of a String
It is the number of symbols present in a string (Denoted by |S|)
Examples : If S = ‘cabcad’, |S|= 6
If |S|= 0, it is called an empty string (Denoted by λ or ε)
 Related Terminologies
• Kleene Star
The Kleene star, ∑*, is a unary operator on a set of symbols or strings, ∑, that
gives the infinite set of all possible strings of all possible lengths
over ∑ including λ
Representation: ∑* = ∑0 ∪ ∑1 ∪ ∑2 ∪……. where ∑p is the set of all possible
strings of length p
Example: If ∑ = {a, b}, ∑* = {λ, a, b, aa, ab, ba, bb,………..}
• Kleene Closure / Plus
The set ∑+ is the infinite set of all possible strings of all possible lengths over ∑
excluding λ
Representation: ∑+ = ∑1 ∪ ∑2 ∪ ∑3 ∪…….
∑+ = ∑* − { λ }
Example : If ∑ = { a, b } , ∑+ = { a, b, aa, ab, ba, bb,………..}
 Related Terminologies
• Language
A language is a subset of ∑* for some alphabet ∑
It can be finite or infinite
Example :
 If the language takes all possible strings of length 2 over ∑ = {a, b},
then L = { ab, aa, ba, bb }
 Finite Automaton
• Finite Automaton can be classified into two types:
Deterministic Finite Automaton (DFA)
Non-deterministic Finite Automaton (NDFA / NFA)
• Deterministic Finite Automaton (DFA)
In DFA, for each input symbol, one can determine the state to which the machine will
move
 Hence, it is called Deterministic Automaton
 As it has a finite number of states
The machine is called Deterministic Finite Machine or Deterministic Finite
Automaton
 Deterministic Finite Automaton
• A DFA can be represented by a 5-tuple (Q, ∑, δ, q0, F) where:
Q is a finite set of states
∑ is a finite set of symbols called the alphabet
δ is the transition function where δ: Q × ∑ → Q
q0 is the initial state from where any input is processed (q0 ∈ Q)
F is a set of final state/states of Q (F ⊆ Q)
• Graphical Representation of a DFA
A DFA is represented by digraphs called state diagram
The vertices represent the states
The arcs labelled with an input alphabet show the transitions
The initial state is denoted by an empty single incoming arc
The final state is indicated by double circles
 Deterministic Finite Automaton
• Let a deterministic finite automaton be:
Q = {a, b, c},
∑ = {0, 1},
q0 = {a},
F = {c}, and
Transition function δ as shown by the table:
Present State Next State for Input 0 Next State for Input 1
a a b
b c a
c b c
 Deterministic Finite Automaton
• Its graphical representation would be as follows:

• Where the initial state is shown by a node with empty source

incoming arc and final state by a node with double circle.
 Non-deterministic Finite Automaton

• In NDFA, for a particular input symbol, the machine can move to any
combination of the states in the machine
In other words, the exact state to which the machine moves cannot be
determined
Hence, it is called Non-deterministic Automaton
As it has finite number of states, the machine is called Non-deterministic Finite
Machine or Non-deterministic Finite Automaton
 Non-deterministic Finite Automaton
• Formal Definition of an NDFA:
• An NDFA can be represented by a 5-tuple (Q, ∑, δ, q0, F) where:
Q is a finite set of states
∑ is a finite set of symbols called the alphabets
δ is the transition function where δ: Q × ∑ → 2Q
(Here the power set of Q (2Q) has been taken because in case of NDFA, from a state,
transition can occur to any combination of Q states)
q0 is the initial state from where any input is processed (q0 ∈ Q)
F is a set of final state/states of Q (F ⊆ Q)
 Graphical Representation of an NDFA: (same as DFA)
• An NDFA is represented by digraphs called state diagram
 The vertices represent the states
 The arcs labelled with an input alphabet show the transitions
 The initial state is denoted by an empty single incoming arc
 The final state is indicated by double circles
• Example: Let a non-deterministic finite automaton be :
• Q = {a, b, c}
Present State Next State for Input 0 Next State for Input 1
• ∑ = {0, 1}
a a, b b
• q0 = {a}
• F = {c} b c a, c
c b, c c
• The transition function δ as shown in the Table
 Graphical Representation of an NDFA: (same as DFA)
• Its graphical representation would be as follows −
 DFA vs NDFA
Some key differences between DFA and NDFA:
DFA
The transition from a state is to a single particular
next state for each input symbol
Hence it is called deterministic
Empty string transitions are not seen in DFA
Backtracking is allowed in DFA
A string is accepted by a DFA, if it transits to a final
state
NDFA
The transition from a state can be to multiple next
states for each input symbol
 Hence it is called non-deterministic
NDFA permits empty string transitions
In NDFA, backtracking is not always possible
A string is accepted by a NDFA, if at least one of all
possible transitions ends in a final state
 Applications: Acceptors, Classifiers and Transducers
• Acceptor (Recognizer)
An automaton that computes a Boolean function is called an acceptor
 All the states of an acceptor is either accepting or rejecting the inputs given to it
• Classifier
A classifier has more than two final states and it gives a single output when it
terminates
• Transducer
An automaton that produces outputs based on current input and/or previous
state is called a transducer
Transducers can be of two types:
Mealy Machine − The output depends both on the current state and the current
input
Moore Machine − The output depends only on the current state
 Acceptability by DFA and NDFA
• A string is accepted by a DFA/NDFA iff the DFA/NDFA starting at the initial
state ends in an accepting state (any of the final states) after reading the
string wholly:

A string S is accepted by a DFA/NDFA (Q, ∑, δ, q0, F), iff δ*(q0, S) ∈ F

The language L accepted by DFA/NDFA is {S | S ∈ ∑* and δ*(q0, S) ∈ F}
A string S′ is not accepted by a DFA/NDFA (Q, ∑, δ, q0, F), iff δ*(q0, S′) ∉ F
The language L′ not accepted by DFA/NDFA (Complement of accepted
language L) is {S | S ∈ ∑* and δ*(q0, S) ∉ F}
 Acceptability by DFA and NDFA:
Example
• Consider the DFA shown in the Figure
• From the DFA, the acceptable strings
can be derived:
Strings accepted by the DFA: {0, 00, 11,
010, 101, ...........}
Strings not accepted by the DFA: {1, 011,
111, ........}
 NDFA to DFA Conversion
• Problem Statement:
Let X = (Qx, ∑, δx, q0, Fx) be an NDFA
which accepts the language L(X)
We have to design an equivalent
DFA Y = (Qy, ∑, δy, q0, Fy) such
that L(Y) = L(X)
The right-hand procedure converts
the NDFA to its equivalent DFA
Algorithm
Input − An NDFA
Output − An equivalent DFA
Step 1 − Create state table from the given NDFA
Step 2 − Create a blank state table under possible input
alphabets for the equivalent DFA
Step 3 − Mark the start state of the DFA by q0 (Same as the
NDFA)
Step 4 − Find out the combination of States {Q0, Q1,... , Qn} for
each possible input alphabet
Step 5 − Each time we generate a new DFA state under the
input alphabet columns, we have to apply step 4 again,
otherwise go to step 6
Step 6 − The states which contain any of the final states of the
NDFA are the final states of the equivalent DFA
 NDFA to DFA Conversion: Example
• Consider the NDFA shown
in the figure
q δ(q,0) δ(q,1)
a {a,b,c,d,e} {d,e}
b {c} {e}
c ∅ {b}
d {e} ∅
e ∅ ∅
 NDFA to DFA Conversion: Example
q δ(q,0) δ(q,1)
• Using the above
algorithm, we find its [a] [a,b,c,d,e] [d,e]
equivalent DFA [a,b,c,d,e] [a,b,c,d,e] [b,d,e]
• The state table of the [d,e] [e] ∅
DFA is shown in the right
[b,d,e] [c,e] [e]
[e] ∅ ∅
[c, e] ∅ [b]
[b] [c] [e]
[c] ∅ [b]
 NDFA to DFA Conversion: Example

• The state diagram of the

DFA is as on the right
 Turing Machine
• A Turing Machine (TM) is a mathematical model
(computing device) which consists of an infinite
length tape divided into cells on which input is given
• It consists of a head which reads the input tape. A
state register stores the state of the Turing machine
• After reading an input symbol, it can replace with
another symbol, its internal state is changed, and it
moves from one cell to the right or left
• If the TM reaches the final state, the input string is
accepted, otherwise rejected
 Turing Machine
• TM consists of a :
Tape – divided into cells, infinite, a cell can hold a tape symbol or blank symbol (#)
Finite control – At a particular instant , finite control is in a state. States are divided
into
 Start state (q0)
 Halt state (h)
 Intermediate states (q1, q2, ...)
Tape head – points to one of the tape cells. It communicates between finite control
and tape. It can move left, right or remain stationary
 Turing Machine
• A TM can be formally described as a 7-tuple (Q, ∑, Γ, δ, q0, #, h) where:

Q is a finite set of states

∑ is the input alphabet/symbols
Γ is the tape alphabet/symbols including #
δ is a transition (next move) function; δ : (Q × Γ) → (Q × Γ × {L, R, N})
q0 is the initial (start) state
# is the blank symbol
h is the set of final (halt) states
 Turing Machine
Example:
• The TM is in state q and head points to tape
symbol c
• Let a transition be defined as δ : (q,c) =
(q1,b,R)
That is, if TM is in state q and points to input
symbol, c, then it enters state, q1, replaces
symbol , c with b and moves one position
towards right (R)
• Thus the TM now is as in the next slide
 Turing Machine
• Now the TM is in state q1 and head points
to the tape symbol a
• Let a move of the TM be defined as as δ :
(q1,a) = (q2,d,L)
That is if TM is in state q1 and points to input
symbol, a, then it enters state q2, replaces
symbol, a , with, d, and moves one position
towards left (L)
• Thus the TM now is as in next slide
 Turing Machine
• Now the TM is in state q2, and head points
to the tape symbol, b
• Let a move of the TM be defined as δ :
(q2,b) = (q1,y,N)
• That is, if TM is in state q2 and points to input
symbol, b, then it enters state, q1, replaces
symbol, b, with y and head remains in the
same position (N)
• Thus the TM is now as in next slide
Turing Machine
 Language Acceptability by TM
• A string w is said to be accepted by a Turing
machine if TM halt in an accepting state
• A string w is not accepted by a TM, if TM halts in
non-accepting state or does not halt
• Example: Consider a TM on the right:
 Language Acceptability by TM
• Check whether a string 011 is accepted by
the above TM?
Initially, TM is in state q1, and the tape contain
the given string 011
 Initially head points to symbol 0 in the input
string
Language Acceptability by TM
Language Acceptability by TM
 Representation of Turing Machines
• We can represent a TM in different ways
Instantaneous descriptions
Transition tables
Transition diagrams
• An instantaneous description, provide an
instant of TM (as on the right) and
descriptions as below:
Representation of TM by Instantaneous Descriptions ...
Let a transition be defined as:
 Representation of TM by Transition Table
• Let a Turing machine be defined as
on the right:
 Representation of TM by Transition Table
• The TM can be represented as
Transition table on the right
• In the transition table, the
entry (q2,y,R) corresponding to
row q2 and column y denotes
the transition
δ : (q2,y) = (q2,y,R)
 Representation of TM by Transition Diagram
• Let TM be defined as on the right:
 Representation of TM by Transition Diagram
• The transition diagram
corresponding to the above TM is
as shown on the right:
• In the diagram, part of the
picture:
 Language Decidability
• A language is called Decidable or Recursive if there is a Turing
machine which accepts and halts on every input string w
• Every decidable language is Turing-Acceptable
 Language Decidability
• A decision problem P is decidable if the language L of all yes instances to P is
decidable
• For a decidable language, for each input string, the TM halts either at the
accept or the reject state as depicted in the following diagram
 Language Decidability : Example
• Example 1: Find out whether the following problem is decidable or not:
Is a number ‘m’ prime?
• Solution :
Prime numbers = {2, 3, 5, 7, 11, 13, …………..}
Divide the number ‘m’ by all the numbers between ‘2’ and ‘√m’ starting from ‘2’
If any of these numbers produce a remainder zero, then it goes to the “Rejected
state”, otherwise it goes to the “Accepted state”
So, here the answer could be made by ‘Yes’ or ‘No’
Hence, it is a decidable problem
 Language Decidability : Example
• Example 2 : Given a regular language L and string w, how can we check if w ∈ L?
• Solution : Take the DFA that accepts L and check if w is accepted
 Un-decidable Languages
• For an undecidable language, there is no
Turing Machine which accepts the language
and makes a decision for every input
string w (TM can make decision for some
input string though)
• A decision problem P is called “undecidable”
if the language L of all yes instances to P is
not decidable
• Undecidable languages are not recursive
languages, but sometimes, they may be
recursively enumerable languages
Un-decidable Languages : Example
• The halting problem of Turing machine
• The mortality problem
• The mortal matrix problem
• The Post correspondence problem
• etc
 Automata and Turing Machine
• The figure on the right depict
languages recognizable under
different computing machines
(automata)
• The complex a language the more its
expressive power:
• Unrestricted grammars produce
unrestricted languages (type-o
languages)
• Type-o languages are recognized using
Turing machines
 Tractability
• Some problems are intractable:
 As they grow large, we are unable to solve them in reasonable time
• What constitutes reasonable time?
 Standard working definition: polynomial time
 On an input of size n the worst-case running time is O(nk) for some constant k
• O(n2), O(n3), O(1), O(n log n), O(2n), O(nn), O(n!)
 Polynomial time: O(n2), O(n3), O(1), O(n log n)
 Not in polynomial time: O(2n), O(nn), O(n!)
 Knapsack Problem
• Example of a one-dimensional (constraint)
knapsack problem:
Which boxes should be chosen to maximize the amount
of money while still keeping the overall weight under or
equal to 15 kg
A multiple constrained problem could consider both
the weight and volume of the boxes
• Solution:
If any number of each box is available, then three
yellow boxes and three grey boxes; if only the shown
boxes are available, then all but the green box
 Fractional Knapsack Problem
• Given weights and values of n items, we need to put these items in a knapsack of
capacity W to get the maximum total value in the knapsack:
 In the 0-1 Knapsack problem, we are not allowed to break items: either take the whole item
or don’t take it
 In Fractional Knapsack, we can break items for maximizing the total value of knapsack ---
thus called the fractional knapsack problem
• Example:
Input:
Items as (value, weight) pairs: arr[] = {{60, 10}, {100, 20}, {120, 30}}
Knapsack Capacity, W = 50;

Output:
Maximum possible value = 240, by taking items of weight 10 and 20 kg and 2/3 fraction of 30
kg. Hence total price will be 60 + 100 + (2/3)(120) = 240
 Fractional Knapsack Problem

• A brute-force solution would be to try all possible subset with all different
fraction but that will be too much time taking
• An efficient solution may make use of Greedy approach
The basic idea of the greedy approach is to calculate the ratio value/weight for each
item and sort the item on basis of this ratio
 Then take the item with the highest ratio and add them until we can’t add the next item
as a whole and at the end add the next item as much as we can
 Which will always be the optimal solution to this problem
• As main time taking step is sorting, the whole problem can be solved in O(n log n)
Travelling Salesman and Hamiltonian Cycle Problems
• A Hamiltonian cycle (also tour, circuit) :
A cycle visiting each vertex exactly once
Graphs are said to be Hamiltonian if they contain a Hamiltonian cycle
• A Hamiltonian path
A path visiting each vertex exactly once
• The decision problems ask whether a Hamiltonian cycle or path exists in
a given graph
• The goal of Travelling salesman problem (TSP):
Find the minimum Hamiltonian cycle (Min-HC) i.e., a cycle that visits each city
once and exactly once and incurs the least length, time or cost, etc
• It has been proven to be NP-complete - There is no exact polynomial algorithm
for TSP until NP=P
 Polynomial-Time Algorithms
• Are some problems solvable in polynomial time?
Of course: many algorithms we’ve studied provide polynomial-time solutions to
some problems
• Are all problems solvable in polynomial time?
 No: Turing’s “Halting Problem” is not solvable by any computer, no matter how
much time is given
• Most problems that do not yield polynomial-time algorithms are either
optimization or decision problems
 Optimization/Decision Problems
• An optimization problem tries to find an optimal solution
• A decision problem tries to answer a yes/no question
• Many problems will have decision and optimization versions
 Example: Travelling salesman problem
 optimization: find hamiltonian cycle of minimum weight
 decision: is there a hamiltonian cycle of weight ≤ k
• Some problems are decidable, but intractable:
As they grow large, we are unable to solve them in reasonable time
Is there a polynomial-time algorithm that solves the problem?
 The Class P
• P:
The class of decision problems that have polynomial-time deterministic algorithms
That is, they are solvable in O(p(n)), where p(n) is a polynomial on n
 A deterministic algorithm is (essentially) one that always computes the correct
answer
• Why polynomial?
 if not, very inefficient
 Nice closure properties: the sum and composition of two polynomials are always
polynomials too
 Sample Problems in P

• Fractional Knapsack
• MST
• Sorting
• Others?
 The class NP
• NP:
The class of decision problems that are solvable in polynomial time on a
nondeterministic machine (or with a nondeterministic algorithm)
 A determinstic computer is what we know
A nondeterministic computer is one that can “guess” the right answer or solution
 Think of a nondeterministic computer as a parallel machine that can freely spawn an infinite
number of processes
• Thus NP can also be thought of as:
The class of problems whose solutions can be verified in polynomial time
Note that NP stands for “Nondeterministic Polynomial-time”
 The class NP
• If problem A is reducible to problem B it is no harder than B
• Conversely, if A is known to be hard then reducing it to B shows that B is at least as hard:
 B serves as an upper bound on the complexity of A, while A serves as a lower bound on the complexity of B
• A problem is said to be NP-complete if any problem in the class NP can be reduced to it and it is
therefore as hard as any problem in NP
• The class of NP complete problems includes
 Satisfiability (for Boolean expressions)
 Hamiltonian path problem
 Travelling salesman problem
• An NP-complete problem “represents” the class NP in the sense that its solution in polynomial
time implies P=NP
• The strong evidence that P is not equal to NP for a wide range of models of computing includes
the fact that no NP-complete problem has yet been solved in polynomial time
• Complexity metrics are remarkably robust with respect to alternative models of computation,
but if unlimited (exponential) parallelism is permitted, then P=NP
 Review: P and NP Summary
• P = set of problems that can be solved in polynomial time
 Examples: Fractional Knapsack, …
• NP = set of problems for which a solution can be verified in
polynomial time
 Examples: TSP, Hamiltonian Cycle, ...
• Clearly P ⊆ NP
• Open question: Does P = NP?
 Most suspect not
An August 2010 claim of proof that P ≠ NP, by Vinay Deolalikar, researcher at
HP Labs, Palo Alto, has flaws
 Review: NP-complete Problems
• A decision problem D is NP-complete iff
1. D ϵ NP
2. Every problem in NP is polynomial-time reducible to D

• “NP-Complete” comes from:

 Nondeterministic Polynomial Complete - “Solve one, Solve them all”
• There are more NP-Complete problems than provably intractable
problems