You are on page 1of 33

Theory of Computation

Dr. Sarmad Abbasi

Dr. Sarmad Abbasi ()

Theory of Computation

1 / 33

Lecture 20: Overview

Incompressible strings
Minimal Length Descriptions
Descriptive Complexity

Dr. Sarmad Abbasi ()

Theory of Computation

2 / 33

Incompressible strings

Recall that a string, x, is b-compressible if


K (x) |x| b
1

A string is b-incompressible if it is not b-compressible.

A string is incompressible if it is not 1-compressible.

Incompressible strings cannot be compressed. Their description is at least


as long as the string itself.

Dr. Sarmad Abbasi ()

Theory of Computation

3 / 33

Incompressible strings

Incompressible strings look like random strings.

This can be made precise in many ways.

Many properties of random strings are also true for incompressible strings.
For example

Dr. Sarmad Abbasi ()

Theory of Computation

4 / 33

Incompressible strings

We proved two theorems:

Theorem
For every n there is a incompressible string of length n.

Theorem
The number of b-incompressible strings of length b is at least
2n 2nb+1 + 1
The proof is simple counting!

Dr. Sarmad Abbasi ()

Theory of Computation

5 / 33

Incompressible strings

Incompressible strings have roughly the same number of 0s and 1s.

The longest run of 0 is of size O(log n).

Let us prove a theorem which is central in this theory.

Dr. Sarmad Abbasi ()

Theory of Computation

6 / 33

Incompressible strings

Theorem
A computable property that holds for almost all strings also holds for
incompressible strings.
Let us formalize this statement. A property p is a mapping from set of all
strings to {TRUE , FALSE }. Thus
p : {0, 1} {true, false}

Dr. Sarmad Abbasi ()

Theory of Computation

7 / 33

Incompressible strings

We say that p holds for almost all strings if


|{x {0, 1}n : p(x) = false }|
2n
approaches 0 as n approaches infinity.

Dr. Sarmad Abbasi ()

Theory of Computation

8 / 33

Incompressible strings

Theorem
Let p be a computable property. If p holds for almost all strings the for
any b > 0, the property p is false on only finitely many strings that are
incompressible by b.
Let M be the following algorithm:
1

On input i (in binary)

Find the ith string s such that p(s) = False. Considering the strings
lexicographically.

Output s.

Dr. Sarmad Abbasi ()

Theory of Computation

9 / 33

Incompressible strings

We can use M to obtain short descriptions of string that do not have


property p. Let x be a string that does not have property p. Let ix be the
index of x in the list of all strings that do not have property p ordered
lexicographically. Then hM, ix i is a description of x. The length of this
description is
|ix | + c

Dr. Sarmad Abbasi ()

Theory of Computation

10 / 33

Incompressible strings

Now we count. For any give b > 0 select n such that at most
1
2b+c
fraction of the strings of length n do not have property p. Note that
since p holds for almost all strings, this is true for all sufficiently large n.

Dr. Sarmad Abbasi ()

Theory of Computation

11 / 33

Incompressible strings

Then we have
ix

2n

2b+c
Therefore, |ix | n b c. Hence the length of hMiix is at most
n b c + c = n b. Thus
K (x) n b
which means x is b-compressible. Hence, only finitely many strings that
fail p can be incompressible by b.

Dr. Sarmad Abbasi ()

Theory of Computation

12 / 33

Incompressible strings

We can ask a very interesting question now:


Is it possible to compress a string that is already compressed?
The answer to this question seems to be no! Let us prove something that
is quite close to this.

Dr. Sarmad Abbasi ()

Theory of Computation

13 / 33

Incompressible strings

Theorem
There is a constant b such that for every string x the minimal description
d(x) is incompressible by b.
Roughly the idea of the proof is that
If d(x) were compressible we could use the compressed version of d(x) to
obtain a smaller description of x than d(x).

Dr. Sarmad Abbasi ()

Theory of Computation

14 / 33

Incompressible strings
Consider the following TM M:
1

On input hR, y i

Run R on y and reject if its output is not of the form hS, zi

Run S on z and halt with its output on the tape.

This machine is decoding twice.


Let b = 2|hMi| + 3. We now show that b satisfies the statement of the
theorem. Suppose on the contrary that d(x) is b compressible for some
string. The
|d(d(x))| |d(x)| b.
But what happens if we run M on d(d(x)). Note that it outputs
x

Dr. Sarmad Abbasi ()

Theory of Computation

15 / 33

Incompressible strings

Thus < M > d(d(x)) is a description of x. How long is this description?


2|hMi| + 2 + |d(d(x))| = (b 1) + (|d(x) b) = |d(x)| 1
This contradicts the fact that d(x) is the minimal description of x.

Dr. Sarmad Abbasi ()

Theory of Computation

16 / 33

Complexity Theory

Now we will look at resource bounded computations.


Two of the most important resources are going to be
1

Time

Space

Dr. Sarmad Abbasi ()

Theory of Computation

17 / 33

Complexity Theory
Let
A = {0k 1k : k 0}
Consider the following TM that accepts A
1

Scan across the tape to check if all 0s are before 1s if not reject.

Scan across the tape crossing of a 0 and a 1 each time.

If 0s remain and 1s finished reject.

if 1s remain and 0s are finished reject.

if both 0s and 1s are finished accept.

Goto Step 2

We want to analyze this algorithm and talk about how many steps (time)
does it take.

Dr. Sarmad Abbasi ()

Theory of Computation

18 / 33

Complexity Theory
A step of the Turing machine is assumed to take unit time. Lets start with
a definition. Let M be a TM that halts on all inputs.
The running time or the time complexity of M is a function
f :NN
such that f (n) is the maximum number of steps that M uses on any input
of length n.
1

We say that M runs in time f (n).

M is f (n)-time TM.

Dr. Sarmad Abbasi ()

Theory of Computation

19 / 33

Big O notation

We want to understand how much time a TM takes in the worst case.


The exact running time t(n) is going to be very complicated expression. In
some cases we will not be able to compute it.
But what we are interested in is getting a rough idea of how t(n) grows for
large inputs. So we will use asymptotic notation.

Dr. Sarmad Abbasi ()

Theory of Computation

20 / 33

Big O notation

Lets look at an example let


f (n) = 5n3 + 13n2 6n + 48
This function has four terms. He most important term is
5n3
which will be the bulk if n is large.

Dr. Sarmad Abbasi ()

Theory of Computation

21 / 33

Big O notation

In fact
5n3
the constant 5 does not contribute to the growth of this function. Thus
we will say that f (n) is order n3 . We will denote this by
f (n) = O(n3 ).
Let us state this precisely:

Dr. Sarmad Abbasi ()

Theory of Computation

22 / 33

Big O notation

Let f , g : N R + . We say that


f (n) = O(g (n))
if there are positive integers c, n0 such that for all n n0
f (n) cg (n)
When f (n) = O(g (n)) we say that g is an asymptotic upper-bound on f .
Sometime we will just say that g is an upper bound on f .

Dr. Sarmad Abbasi ()

Theory of Computation

23 / 33

Big O notation

Intuitively if f (n) = O(g (n)) you can think that f is less than or equal to
some multiple of g . For most functions the highest order term is is an
obvious candidate to be g .

Dr. Sarmad Abbasi ()

Theory of Computation

24 / 33

Big O notation

Let
f (n) = 13n4 + 7n2 + 13
Then
f (n) = O(n4 )
Furthermore,
f (n) = O(n5 )
But it is not true that
f (n) = O(n3 )

Dr. Sarmad Abbasi ()

Theory of Computation

25 / 33

Big O notation

When we say that f (n) = O(g (n)) we are only claiming that g (n) is an
asymptotic upper bound on f . It may not be the best one. Lets look at
other examples.

Dr. Sarmad Abbasi ()

Theory of Computation

26 / 33

Big O notation
Suppose that
f (n) = logb n
where b is a constant. Note that
logb n =

log2 n
.
log2 b

Hence we can say


f (n) = O(log n)
Let
f2 (n) = 5n log n + 200n log log n + 17
then
f2 (n) = O(n log n)

Dr. Sarmad Abbasi ()

Theory of Computation

27 / 33

Big O notation
We can use O notation in exponents also. Thus
2O(n)
stands for a function that is upper bounded by
2cn
for some constant c. We have to be careful though
For example
2O(log n)
Well remember that O(log n) has a constant attached to it. Thus this is
upper bounded by
2c log n
by some constant c. Now, we get
2c log n = (2log n )c = nc
Dr. Sarmad Abbasi ()

Theory of Computation

28 / 33

Big O notation

Bounds of the form nc are called polynomial bounds. Where as bounds of


c
the form 2n are called exponential bounds.

Dr. Sarmad Abbasi ()

Theory of Computation

29 / 33

small o notation

The Big O notation roughly says that f is upper bounded by g . Now we


want to say
f grows strictly slower that g .
Let f , g : N R + . We say that f (n) = o(g (n)) if
f (n)
=0
n g (n)
lim

Thus the ratio of f and g becomes closer and closer to 0.

Dr. Sarmad Abbasi ()

Theory of Computation

30 / 33

small o notation

n = o(n)

n = o(n log log n)

n log log n = o(n log n)

n log n = o(n2 )

n2 = o(n3 ).

Dr. Sarmad Abbasi ()

Theory of Computation

31 / 33

small o notation

Lets look at a new TM that recognizes


A = {0k 1k : k 0}
1

Scan the tape and make sure all the 0s are before 1s.

Repeat till there are 0s and 1s on the tape.

Scan the tape and if the total number of 0s and 1s is odd reject.

Scan and cross of every other 0 and do the same with every other 1.

If all 0s and 1s are crossed accept.

Thus if we have 13 0s in the next stage there would be 6.

Dr. Sarmad Abbasi ()

Theory of Computation

32 / 33

small o notation

How much time does this TM take?


O(n log n)
Thus the time taken by this TM is o(n2 ). Hence, this reflects a real
algorithmic improvement.

Dr. Sarmad Abbasi ()

Theory of Computation

33 / 33