You are on page 1of 33

Theory of Computation

Theory of Computation

1 / 33

Lecture 20: Overview

Incompressible strings
Minimal Length Descriptions
Descriptive Complexity

Theory of Computation

2 / 33

Incompressible strings

K (x) |x| b
1

Incompressible strings cannot be compressed. Their description is at least

as long as the string itself.

Theory of Computation

3 / 33

Incompressible strings

This can be made precise in many ways.

Many properties of random strings are also true for incompressible strings.
For example

Theory of Computation

4 / 33

Incompressible strings

We proved two theorems:

Theorem
For every n there is a incompressible string of length n.

Theorem
The number of b-incompressible strings of length b is at least
2n 2nb+1 + 1
The proof is simple counting!

Theory of Computation

5 / 33

Incompressible strings

Let us prove a theorem which is central in this theory.

Theory of Computation

6 / 33

Incompressible strings

Theorem
A computable property that holds for almost all strings also holds for
incompressible strings.
Let us formalize this statement. A property p is a mapping from set of all
strings to {TRUE , FALSE }. Thus
p : {0, 1} {true, false}

Theory of Computation

7 / 33

Incompressible strings

We say that p holds for almost all strings if

|{x {0, 1}n : p(x) = false }|
2n
approaches 0 as n approaches infinity.

Theory of Computation

8 / 33

Incompressible strings

Theorem
Let p be a computable property. If p holds for almost all strings the for
any b > 0, the property p is false on only finitely many strings that are
incompressible by b.
Let M be the following algorithm:
1

On input i (in binary)

Find the ith string s such that p(s) = False. Considering the strings
lexicographically.

Output s.

Theory of Computation

9 / 33

Incompressible strings

We can use M to obtain short descriptions of string that do not have

property p. Let x be a string that does not have property p. Let ix be the
index of x in the list of all strings that do not have property p ordered
lexicographically. Then hM, ix i is a description of x. The length of this
description is
|ix | + c

Theory of Computation

10 / 33

Incompressible strings

Now we count. For any give b > 0 select n such that at most
1
2b+c
fraction of the strings of length n do not have property p. Note that
since p holds for almost all strings, this is true for all sufficiently large n.

Theory of Computation

11 / 33

Incompressible strings

Then we have
ix

2n

2b+c
Therefore, |ix | n b c. Hence the length of hMiix is at most
n b c + c = n b. Thus
K (x) n b
which means x is b-compressible. Hence, only finitely many strings that
fail p can be incompressible by b.

Theory of Computation

12 / 33

Incompressible strings

We can ask a very interesting question now:

Is it possible to compress a string that is already compressed?
The answer to this question seems to be no! Let us prove something that
is quite close to this.

Theory of Computation

13 / 33

Incompressible strings

Theorem
There is a constant b such that for every string x the minimal description
d(x) is incompressible by b.
Roughly the idea of the proof is that
If d(x) were compressible we could use the compressed version of d(x) to
obtain a smaller description of x than d(x).

Theory of Computation

14 / 33

Incompressible strings
Consider the following TM M:
1

On input hR, y i

This machine is decoding twice.

Let b = 2|hMi| + 3. We now show that b satisfies the statement of the
theorem. Suppose on the contrary that d(x) is b compressible for some
string. The
|d(d(x))| |d(x)| b.
But what happens if we run M on d(d(x)). Note that it outputs
x

Theory of Computation

15 / 33

Incompressible strings

Thus < M > d(d(x)) is a description of x. How long is this description?

2|hMi| + 2 + |d(d(x))| = (b 1) + (|d(x) b) = |d(x)| 1
This contradicts the fact that d(x) is the minimal description of x.

Theory of Computation

16 / 33

Complexity Theory

Now we will look at resource bounded computations.

Two of the most important resources are going to be
1

Time

Space

Theory of Computation

17 / 33

Complexity Theory
Let
A = {0k 1k : k 0}
Consider the following TM that accepts A
1

Scan across the tape to check if all 0s are before 1s if not reject.

if both 0s and 1s are finished accept.

Goto Step 2

We want to analyze this algorithm and talk about how many steps (time)
does it take.

Theory of Computation

18 / 33

Complexity Theory
A step of the Turing machine is assumed to take unit time. Lets start with
a definition. Let M be a TM that halts on all inputs.
The running time or the time complexity of M is a function
f :NN
such that f (n) is the maximum number of steps that M uses on any input
of length n.
1

We say that M runs in time f (n).

M is f (n)-time TM.

Theory of Computation

19 / 33

Big O notation

We want to understand how much time a TM takes in the worst case.

The exact running time t(n) is going to be very complicated expression. In
some cases we will not be able to compute it.
But what we are interested in is getting a rough idea of how t(n) grows for
large inputs. So we will use asymptotic notation.

Theory of Computation

20 / 33

Big O notation

Lets look at an example let

f (n) = 5n3 + 13n2 6n + 48
This function has four terms. He most important term is
5n3
which will be the bulk if n is large.

Theory of Computation

21 / 33

Big O notation

In fact
5n3
the constant 5 does not contribute to the growth of this function. Thus
we will say that f (n) is order n3 . We will denote this by
f (n) = O(n3 ).
Let us state this precisely:

Theory of Computation

22 / 33

Big O notation

Let f , g : N R + . We say that

f (n) = O(g (n))
if there are positive integers c, n0 such that for all n n0
f (n) cg (n)
When f (n) = O(g (n)) we say that g is an asymptotic upper-bound on f .
Sometime we will just say that g is an upper bound on f .

Theory of Computation

23 / 33

Big O notation

Intuitively if f (n) = O(g (n)) you can think that f is less than or equal to
some multiple of g . For most functions the highest order term is is an
obvious candidate to be g .

Theory of Computation

24 / 33

Big O notation

Let
f (n) = 13n4 + 7n2 + 13
Then
f (n) = O(n4 )
Furthermore,
f (n) = O(n5 )
But it is not true that
f (n) = O(n3 )

Theory of Computation

25 / 33

Big O notation

When we say that f (n) = O(g (n)) we are only claiming that g (n) is an
asymptotic upper bound on f . It may not be the best one. Lets look at
other examples.

Theory of Computation

26 / 33

Big O notation
Suppose that
f (n) = logb n
where b is a constant. Note that
logb n =

log2 n
.
log2 b

Hence we can say

f (n) = O(log n)
Let
f2 (n) = 5n log n + 200n log log n + 17
then
f2 (n) = O(n log n)

Theory of Computation

27 / 33

Big O notation
We can use O notation in exponents also. Thus
2O(n)
stands for a function that is upper bounded by
2cn
for some constant c. We have to be careful though
For example
2O(log n)
Well remember that O(log n) has a constant attached to it. Thus this is
upper bounded by
2c log n
by some constant c. Now, we get
2c log n = (2log n )c = nc

Theory of Computation

28 / 33

Big O notation

Bounds of the form nc are called polynomial bounds. Where as bounds of

c
the form 2n are called exponential bounds.

Theory of Computation

29 / 33

small o notation

The Big O notation roughly says that f is upper bounded by g . Now we

want to say
f grows strictly slower that g .
Let f , g : N R + . We say that f (n) = o(g (n)) if
f (n)
=0
n g (n)
lim

Thus the ratio of f and g becomes closer and closer to 0.

Theory of Computation

30 / 33

small o notation

n = o(n)

n log log n = o(n log n)

n log n = o(n2 )

n2 = o(n3 ).

Theory of Computation

31 / 33

small o notation

Lets look at a new TM that recognizes

A = {0k 1k : k 0}
1

Scan the tape and make sure all the 0s are before 1s.

Repeat till there are 0s and 1s on the tape.

Scan the tape and if the total number of 0s and 1s is odd reject.

Scan and cross of every other 0 and do the same with every other 1.

Thus if we have 13 0s in the next stage there would be 6.

Theory of Computation

32 / 33

small o notation

How much time does this TM take?

O(n log n)
Thus the time taken by this TM is o(n2 ). Hence, this reflects a real
algorithmic improvement.