12 views

Original Title: Kolmogorov Complexity

Uploaded by Srijit Sanyal

- 033102_1
- 2 Information&Entropy
- ineq2
- K MAPS
- HW2
- cibcb-2012
- Discrete Mathematics
- MMOW With Formatting
- new map data
- DCT in image processing
- 2017.02.03.0915.Koukoulopoulos
- Concrete Customization of {(Measurable, Driven), (Wake Up, Speed Up), (Custom, Event), (Handle, Hold)} Surround Set and Settling Switch Fashion Flow Outlets of Business Benefit
- Wireless Digital Communications
- Deterministic Random Generator
- Compiler Lab SRM IST
- MATSim for IVABM breakdown Structure.pptx
- 5.pdf
- bagheri2016.pdf
- 2008wymic Team
- 409ex61.ppt

You are on page 1of 33

Crash course

13 January 17 January 2014

12:00 to 14:00

Monday

Probability theory

Uncertainty and coding

Student presentations

27 January 31 January 2014

12:00 to 14:00

Tuesday

The weak law of large numbers

The source coding theorem

Location

ILLC, room F1.15,

Science Park 107, Amsterdam

Wednesday

Random processes

Arithmetic coding

Materials

informationtheory.weebly.com

Thursday

Divergence

Kelly Gambling

Contact

Mathias Winther Madsen

mathias.winther@gmail.com

Friday

Kolmogorov Complexity

The limits of statistics

PLAN

Some combinatorical

preliminaries

Turing machines

Kolmogorov complexity

The universality of

Kolmogorov complexity

The equivalence of

Kolmogorov complexity

and coin flipping entropy

Monkeys with typewriters

PLAN

Some combinatorical

preliminaries:

Factorials

Stirling's approximation

Binomial coefficients

approximation

ABC, ACB, BAC, BCA, CAB, CBA

Notation:

n! == 1 2 3 n

or n factorial.

approximated by Stirling's approximation,

ln(n!) == n ln n n

The error of this approximation grows

slower than linearly.

n

10

20

30

40

50

ln(n!)

15.1

42.3

74.6

110.3

148.5

Stir(n)

13.0

40.0

72.0

107.6

145.6

Sproof:

William Feller: An Introduction to

Probability Theory and its Applications (1950)

There are

24

4 3 ==

==

== 12

2

2!

4!

1

1

2

1

2

1

2

2

1

options is a factor of 2! smaller:

24

4!

==

4

2! 2!

*

== 6

*

*

*

*

*

*

identical objects into n distinct boxes is

n

k

n!

==

(n k)! k!

n choose k.

For a nice introduction, see the fi rst chapter of

Victor Bryant: Aspects of Combinatorics (1993)

Stirling's approximation gives

ln

n

k

k

== n H2

n

measured in nats (natural units, 1.44 bits).

The error grows slower than a linearly.

Example: n = 40, k = 20

137846528820

log

== 25.649 27.726.

1099511627776

PLAN

Some combinatorical

preliminaries

Turing machines

Kolmogorov complexity

The universality of

Kolmogorov complexity

The equivalence of

Kolmogorov complexity

and coin flipping entropy

Monkeys with typewriters

Shortest description?

001001001001.

1100111100111111001111111100.

01011011101111011111.

0100011011000001010011100101110111.

101100010111001000010111111101.

is the length of the shortest program which will

print that string.

001001001001.

1100111100111111001111111100.

01011011101111011111.

0100011011000001010011100101110111.

101100010111001000010111111101.

Theorem:

There are universal machines.

Sproof:

Consequence:

The Kolmogorov complexity of a string

on two different universal machines

differs only by the length of the

longest simulation program:

KM1(x) KM2(x) == c(M1, M2)

(And constants are sublinear.)

the length of the shortest program which will

print that string.

001001001001001001 001001001.

11001111001111110011111111 00.

01011011101111011111 11111111.

010001101100000101001110010 1.

10110001011100100001011111 01.

Theorem:

Most strings don't have any structure.

Proof:

There are 2n strings of length n, and

1 + 2 + 4 + 8 + 16 + + 2n 1

programs of length < n.

PrintString(n, k, i):

construct all string of length n

select the ones containing k 1s

print the ith of those strings.

n = 10,

k = 3,

i = 13.

1 0 1 0 , 1 1 , 1 1 0 1

11 00 11 00 01 11 11 01 11 11 00 11

n = 10,

k = 3,

i = 13.

1 0 1 0 , 1 1 , 1 1 0 1

11 00 11 00 01 11 11 01 11 11 00 11

2 + 4 + 2

coeffi cient is

ln

n

k

k

== n H2

n

measured in nats (natural units, 1.44 bits).

The error grows slower than a linearly.

So

K(x)

k

2n H2

n

where is sublinear.

(n)

complexity is equal to Shannon entropy,

plus or minus a sublinear term.

For other sequences, Kolmogorov complexity is smaller than the Shannon entropy of

the string if modeled as as a coin fl ipping

sequence.

Random

monkey

Universal

machine

Finite string

(or nothing)

inference, Information and control, 1964.

machine you use.

close to 2K(x).

For long strings, fi nding the probability of

the shortest description is thus as good as

summing up the probability of all

descriptions.

- 033102_1Uploaded bysankha
- 2 Information&EntropyUploaded byAbdullah Salem
- ineq2Uploaded bythelastairanand
- K MAPSUploaded bysivar22
- HW2Uploaded byHayden Chappelear-Robbins
- cibcb-2012Uploaded bychris
- Discrete MathematicsUploaded byRonald U. Wacas
- MMOW With FormattingUploaded byHudaSEl-zerii
- new map dataUploaded byragup70
- DCT in image processingUploaded byRoshaney Jkd
- 2017.02.03.0915.KoukoulopoulosUploaded bysqnrdi
- Concrete Customization of {(Measurable, Driven), (Wake Up, Speed Up), (Custom, Event), (Handle, Hold)} Surround Set and Settling Switch Fashion Flow Outlets of Business BenefitUploaded bysfofoby
- Wireless Digital CommunicationsUploaded byGetachew Melese(Ph.D.)
- Deterministic Random GeneratorUploaded bycarlosnestor
- Compiler Lab SRM ISTUploaded byAbhilash Rammstein Pal Cobain
- MATSim for IVABM breakdown Structure.pptxUploaded byKỳ Trí Nguyễn
- 5.pdfUploaded bySivakumar
- bagheri2016.pdfUploaded byMauricio Ferrer
- 2008wymic TeamUploaded byJL
- 409ex61.pptUploaded byAntariksh Shahwal
- UG_CSE_Syllabus.pdfUploaded byllanelli.gracia
- 18MVD0071_ACA_DA-2Uploaded byAkshay Patharkar
- 17728_block Diagram Algebra UploadedUploaded byshubhamgautam56_6497
- Updated Room NumbersUploaded bySri Mathi
- hw1sol.pdfUploaded byPei Jing
- 08-l1stable.pdfUploaded byAshwani Singh
- matej1Uploaded byRax Uta
- Resume 2017 EditUploaded byAnonymous HPyjd21
- 04721720Uploaded byG Venkata Nagesh Kumar
- MELJUN_CORTES_2005 Csci09 Discrete Structures 2Uploaded byMELJUN CORTES, MBA,MPA

- 1224-7958-1-PBUploaded bySrijit Sanyal
- UCB104Uploaded bySrijit Sanyal
- BERCI-2.2Uploaded bySrijit Sanyal
- 0701092Uploaded bySrijit Sanyal
- Lecture 8Uploaded bySrijit Sanyal
- panpsychismUploaded bySrijit Sanyal
- Js SecretsUploaded byPeriyakaruppan Nagarajan
- diffalg1Uploaded bySrijit Sanyal
- fea-magidUploaded bySrijit Sanyal
- Diff FormsUploaded bySrijit Sanyal
- Ehrenfest's TheoremUploaded bySrijit Sanyal
- Waist-hip ratio and cognitive ability: is gluteofemoral fat a privileged store of neurodevelopmental resources?Uploaded byCromiron
- Pistone ExpUploaded bySrijit Sanyal
- Entropy Entanglement Figured 01Uploaded bySrijit Sanyal
- comprehensive world historyUploaded bySuman Hazarika
- Berker Norm Insignif NeuroUploaded bySrijit Sanyal
- Calculus for Mathematicians - DJ Bernstein (12 pgs).pdfUploaded byKim Fierens
- Leo Breiman - Statistical Modeling - Two CulturesUploaded bymurilove
- Mathematica Programming - Advanced IntroUploaded byTraderCat Solaris
- Re WritingUploaded bySrijit Sanyal
- 0901.0952Uploaded bySrijit Sanyal
- Iraq the VoteUploaded bySrijit Sanyal
- Rosen LichtUploaded bySrijit Sanyal
- Overload SmallUploaded bySrijit Sanyal
- 1111.3846Uploaded bySrijit Sanyal
- WassermanUploaded bySrijit Sanyal
- How to Study EconomicsUploaded bySrijit Sanyal
- recounting the rationalsUploaded bySrijit Sanyal
- hl13nflasdgasdgUploaded bySrijit Sanyal

- Blast OverviewUploaded byaderjmc
- Wilf Zeilberger PairUploaded byyossi1234
- CAD CAM Question BankUploaded byrsdeshmukh
- Calculus 04-Extrem ValueUploaded bybudi_umm
- DENAVIT-HARTENBERGUploaded byAnonymous 834jcl1l4
- Report Roller ChainUploaded byPejuangHati
- DocumentUploaded byDrizzle Ventura
- Bingham Fluid Flow in a Plane Narrow slabUploaded byrachma tia
- QP - M1 June 2014Uploaded byMiracle
- Chapter Three 1Uploaded byAnnisa Rizqia
- Circulat Motion Problems.docxUploaded bySadiki Wallace
- supp10Uploaded byGilberto Ruiz
- Http- 139.165.65.75 Projets Biblio Upload Rigo Web Publication009Uploaded byMohamed Fouad
- Atomic Feng Shui Aetheric Diodes and MagnetizationUploaded byDante
- manual-12.12Uploaded byAnanias B Alencar
- d 2363 – 79 r95 _rdiznjmtnzlsotvfmq_Uploaded byalienz1988new
- Tribology - Solid & Semi-Solid LubrificantsUploaded byJorge Vieira
- ECV5701 Notes for Concrete Inspection and Assessment-1Uploaded byahmed almhjani
- Vvvf Inverter Mfc 20 30 Teil2 ThyssenUploaded byWorldtech Ascensores
- Section Properties L Shapes Equal LegsUploaded byjdfdferer
- Algebra Extra Credit Worksheet--Rotations and TransformationsUploaded byGambit King
- 10 Abaqus Vibrations Tutorial (1)Uploaded byAkshay Badani
- 10020688Uploaded bydahaimama
- 822_pdfUploaded byVaidya Nur
- CHAPTER 2.docxUploaded bydinesh naik
- Chaika lenses - part 1.pdfUploaded byJoe Green
- (DEsign Manual) Hollow Section Structural StabilityUploaded byShaileshRastogi
- Google BoasUploaded bytercopy
- Flat Slabs HandoutsUploaded byYasonsky Captain
- PhysicsUploaded byJohn Aldren Molina