Theory of Computation

Introduction
What is the Theory of Computation?
Tries to answer the following questions:

• What are the mathematical properties of

computer hardware and software?
• What is a computation, and what is an
algorithm?
• What are the limitations of computers? Can
“everything” be computed? (As we proceed,
the answer to this question is “no”.)
2
Introduction(Cont’d)

• Central Question in the Theory of

Computation: What are the fundamental
capabilities and limitations of
computers?

3
Introduction(Cont’d)

• Nowadays, the Theory of Computation

can be divided into the following three
areas: Complexity Theory,
Computability Theory, and Automata
Theory.

4
Complexity Theory
• Central Question in Complexity
Theory: Classify problems according
to their degree of “difficulty”. Give
a rigorous proof that problems that
seem to be “hard” are really “hard”.

In complexity theory, the objective is to

classify problems as easy ones and hard
ones;
5
Computability Theory

• Central Question in Computability

Theory: Classify problems as being
solvable or unsolvable.

6
Automata Theory
• Automata Theory deals with
definitions and properties of different
types of “computation models”.

Central Question in Automata Theory: Do

these models have the same power, or can
one model solve more problems than the
other?

7
Models of Computation

8
Computation

CPU memory

9
temporary memory

input
CPU
output

Program memory

10
 3
Example: f ( x)  x

temporary memory

input
CPU
output
Program memory
compute xx
2
compute x x
11
 3
f ( x)  x

temporary memory
input
x2
CPU
output
Program memory
compute xx
2
compute x x
12
 3
temporary memory f ( x)  x
z  2*2  4
f ( x)  z * 2  8
input
x2
CPU
output
Program memory
compute xx
2
compute x x
13
 3
temporary memory f ( x)  x
z  2*2  4
f ( x)  z * 2  8
input
x2
CPU
f ( x)  8
Program memory output
compute xx
2
compute x x
14
 Automaton
temporary memory

Automaton
input
CPU
output

Program memory

15
 Automaton
temporary memory

Automaton
input

output
transition

state

16
 Different Kinds of Automata
Automata are distinguished by the temporary memory

• Finite Automata: no temporary memory

• Pushdown Automata: stack

• Turing Machines: random access memory

17
 Finite Automaton

temporary memory

input
Finite
Automaton
output

Example: Elevators, Vending Machines

(small computing power)
18
 Pushdown Automaton
Temp.
memory Stack Push, Pop

Pushdown input

Automaton
output

Example: Compilers for Programming Languages

(medium computing power)
19
 Turing Machine

Temp.
memory Random Access Memory

input
Turing
Machine
output

Examples: Any Algorithm

(highest computing power)
20
 Power of Automata
Simple More complex Hardest
problems problems problems

Finite Pushdown Turing

Automata Automata Machine

Less power More power

Solve more
computational problems
21
Turing Machine is the most powerful
computational model known

Question: Are there computational

problems that a Turing Machine
cannot solve?

Answer: Yes (unsolvable problems)

22
 What is Complexity Theory?

• Complexity Theory is a central field of the

theoretical foundations of computer science.
• It is concerned with the study of the intrinsic
complexity of computational tasks.

• That is, a typical complexity theoretic study

refers to the computational resources required
to solve a computational task (or a class of such
tasks), rather than referring to a specific
algorithm or an algorithmic schema.

23
What is Complexity Theory?(Cont’d)
• Its “final” goals include the determination
of the complexity of any well defined
computational task.

• Additional goals include:

 obtaining an understanding of the
relations between various computational
phenomena (e.g., relating one fact
regarding computational complexity to
another).
• That is, it provides tools to measure the
difficulty of computational problems both
absolutely and in comparison with other
problems. 24
What is Complexity Theory?(Cont’d)

Generally:
• Absolute Questions:
– How much time is needed to perform
the task?
– How much resources will be needed?
• Relative Questions:
– More difficult than other tasks?
– Are there “most difficult" tasks?

25
What is Complexity Theory?(Cont’d)

• Complexity Theory has failed to

determine the intrinsic complexity of
tasks.
• But it has succeeded in establishing that
two seemingly different computational
tasks are in some sense the same (or,
more precisely, are computationally
equivalent).
…Many relative answers, only few
absolute ones..)
26
Algorithm Design & Analysis Vs Complexity
Theory
• In focusing attention on computational tasks
and algorithms, computability theory has set
the stage for the study of the computational
resources (like time) that are required by
such algorithms.

When this study focuses on the resources

that are necessary for any algorithm that
solves a particular task (or a task of a
particular type), the study becomes part of
the theory of Computational Complexity
(also known as Complexity Theory).
27
 Algorithm Design & Analysis Vs
Complexity Theory(cont’d)

• In contrast, when the focus is on the

design and analysis of specific algorithms
(rather than on the intrinsic complexity of
the task), the study becomes part of a
related subfield that may be called
Algorithmic Design and Analysis.

• …In contrast, Complexity Theory typically

maintains a unity of the study of tasks
solvable within certain resources
(regardless of the origins of these tasks).
28
Algorithm Design & Analysis Vs Complexity
Theory(cont’d)
• When designing an algorithm we “only” need to
develop and analyze one algorithm.

…This provides an upper bound for the

minimal resource requirements with which the
problem can be solved.

• Complexity theory must provide lower bounds for

the minimally necessary resource requirements
that every algorithm that solves the problem
must use.
29
 Algorithm Design & Analysis Vs
Complexity Theory(cont’d)
• To show an upper bound T(n) on the time
complexity of a problem, one needs to show only
that there is a particular algorithm with running
time at most T(n).
• However, proving lower bounds is much more
difficult, since lower bounds make a statement
about all possible algorithms that solve a given
problem.
• To show a lower bound of T(n) for a problem
requires showing that no algorithm can have
time complexity lower than T(n).

30
 Algorithm Design & Analysis Vs
Complexity Theory (cont’d)

• So complexity theory and the design and

analysis of efficient algorithms are the two
areas of computer science which together
fathom the borders between what can and
cannot be done algorithmically with
realistic resource requirements.

31
• Complexity Theory - prove that difficult
problems cannot be solved with modest
resources.

• What are the consequences when we find

out that the problem we are
investigating is not efficiently solvable?

– …We need no longer waste our time

with attempts to obtain an unreachable
goal.
32
• A problem in a very general form with very strict
demands on the quality of the solution.
… If such a general formulation has an efficient
solution… this is great.

…But when this is not the case, specialize the

problem, or perhaps a weaker form of solution will
be accepted.
• In this way we come up with new problems which
are perhaps efficiently algorithmically solvable.
• And so impossibility proofs (negative results)
help us find the problems that are (perhaps “just
barely”) efficiently solvable.
33