5/21/2015

KolmogorovcomplexityWikipedia,thefreeencyclopedia

Kolmogorovcomplexity
FromWikipedia,thefreeencyclopedia

Nottobeconfusedwithdescriptivecomplexitytheory.
Inalgorithmicinformationtheory(asubfield
ofcomputerscienceandmathematics),
theKolmogorovcomplexity(alsoknown
asdescriptive

ThisimageillustratespartoftheMandelbrot
setfractal.Simplystoringthe24bitcolorofeachpixel
inthisimagewouldrequire1.62millionbits,buta
smallcomputerprogramcanreproducethese1.62
millionbitsusingthedefinitionoftheMandelbrotset
andthecoordinatesofthecornersoftheimage.Thus,
theKolmogorovcomplexityoftherawfileencoding
thisbitmapismuchlessthan1.62millionbitsinany
pragmaticmodelofcomputation.

complexity,KolmogorovChaitincomplexity,algorithmicentropy,orprogramsizecomplexity)ofan
object,suchasapieceoftext,isameasureofthecomputabilityresourcesneededtospecifytheobject.It
isnamedafterAndreyKolmogorov,whofirstpublishedonthesubjectin1963.[1][2]
Forexample,considerthefollowingtwostringsof32lowercaselettersanddigits:
abababababababababababababababab

4c1j5b2p0cv4w1x8rx2y39umgw5q85s7
ThefirststringhasashortEnglishlanguagedescription,namely"ab16times",whichconsists
of11characters.Thesecondonehasnoobvioussimpledescription(usingthesamecharacterset)other
thanwritingdownthestringitself,whichhas32characters.
Moreformally,thecomplexityofastringisthelengthoftheshortestpossibledescriptionofthestringin
somefixeduniversaldescriptionlanguage(thesensitivityofcomplexityrelativetothechoiceofdescription
languageisdiscussedbelow).ItcanbeshownthattheKolmogorovcomplexityofanystringcannotbe
data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color%3A

1/5

5/21/2015

KolmogorovcomplexityWikipedia,thefreeencyclopedia

morethanafewbyteslargerthanthelengthofthestringitself.Strings,liketheababexampleabove,
whoseKolmogorovcomplexityissmallrelativetothestring'ssizearenotconsideredtobecomplex.
ThenotionoftheKolmogorovcomplexitycanbeusedtostateandproveimpossibilityresultsakin
toCantor'sdiagonalargument,Gdel'sincompletenesstheorem,andTuring'shaltingproblem.
Contents[hide]
1Definition
2Invariancetheorem
2.1Informaltreatment
2.2Amoreformaltreatment
3Historyandcontext
4Basicresults
4.1UncomputabilityofKolmogorovcomplexity
4.2ChainruleforKolmogorovcomplexity
5Compression
6Chaitin'sincompletenesstheorem
7Minimummessagelength
8Kolmogorovrandomness
9Relationtoentropy
10Conditionalversions
11Seealso
12Notes
13References
14Externallinks

Definition

[edit]

TheKolmogorovcomplexitycanbedefinedforanymathematicalobject,butforsimplicitythescopeofthis
articleisrestrictedtostrings.Wemustfirstspecifyadescriptionlanguageforstrings.Suchadescription
languagecanbebasedonanycomputerprogramminglanguage,suchasLisp,Pascal,orJavavirtual
machinebytecode.IfPisaprogramwhichoutputsastringx,thenPisadescriptionofx.Thelengthofthe
descriptionisjustthelengthofPasacharacterstring,multipliedbythenumberofbitsinacharacter(e.g.
7forASCII).
Wecould,alternatively,chooseanencodingforTuringmachines,whereanencodingisafunctionwhich
associatestoeachTuringMachineMabitstring<M>.IfMisaTuringMachinewhich,oninputw,outputs
stringx,thentheconcatenatedstring<M>wisadescriptionofx.Fortheoreticalanalysis,thisapproachis
moresuitedforconstructingdetailedformalproofsandisgenerallypreferredintheresearchliterature.In
thisarticle,aninformalapproachisdiscussed.
Anystringshasatleastonedescription,namelytheprogram:
functionGenerateFixedString()
returns
data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color%3A

2/5

5/21/2015

KolmogorovcomplexityWikipedia,thefreeencyclopedia

Ifadescriptionofs,d(s),isofminimallength(i.e.itusesthefewestbits),itiscalledaminimal
descriptionofs.Thus,thelengthofd(s)(i.e.thenumberofbitsinthedescription)istheKolmogorov
complexityofs,writtenK(s).Symbolically,
K(s)=|d(s)|.
Thelengthoftheshortestdescriptionwilldependonthechoiceofdescriptionlanguagebuttheeffectof
changinglanguagesisbounded(aresultcalledtheinvariancetheorem).

Invariancetheorem
Informaltreatment

[edit]

[edit]

Therearesomedescriptionlanguageswhichareoptimal,inthefollowingsense:givenanydescriptionof
anobjectinadescriptionlanguage,Icanusethatdescriptioninmyoptimaldescriptionlanguagewitha
constantoverhead.Theconstantdependsonlyonthelanguagesinvolved,notonthedescriptionofthe
object,ortheobjectbeingdescribed.
Hereisanexampleofanoptimaldescriptionlanguage.Adescriptionwillhavetwoparts:
Thefirstpartdescribesanotherdescriptionlanguage.
Thesecondpartisadescriptionoftheobjectinthatlanguage.
Inmoretechnicalterms,thefirstpartofadescriptionisacomputerprogram,withthesecondpartbeing
theinputtothatcomputerprogramwhichproducestheobjectasoutput.
Theinvariancetheoremfollows:GivenanydescriptionlanguageL,theoptimaldescriptionlanguageis
atleastasefficientasL,withsomeconstantoverhead.
Proof:AnydescriptionDinLcanbeconvertedintoadescriptionintheoptimallanguagebyfirst
describingLasacomputerprogramP(part1),andthenusingtheoriginaldescriptionDasinputtothat
program(part2).ThetotallengthofthisnewdescriptionDis(approximately):
|D|=|P|+|D|
ThelengthofPisaconstantthatdoesn'tdependonD.So,thereisatmostaconstantoverhead,
regardlessoftheobjectdescribed.Therefore,theoptimallanguageisuniversaluptothisadditive
constant.

Amoreformaltreatment

[edit]

Theorem:IfK1andK2arethecomplexityfunctionsrelativetoTuringcompletedescription
languagesL1andL2,thenthereisaconstantcwhichdependsonlyonthelanguagesL1andL2chosen
suchthat
s.cK1(s)K2(s)c.
Proof:Bysymmetry,itsufficestoprovethatthereissomeconstantcsuchthatforallstringss
K1(s)K2(s)+c.
data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color%3A

3/5

5/21/2015

KolmogorovcomplexityWikipedia,thefreeencyclopedia

Now,supposethereisaprograminthelanguageL1whichactsasaninterpreterforL2:
functionInterpretLanguage(stringp)
wherepisaprograminL2.Theinterpreterischaracterizedbythefollowingproperty:
Running InterpretLanguage oninputpreturnstheresultofrunningp.
Thus,ifPisaprograminL2whichisaminimaldescriptionofs,then InterpretLanguage (P)returnsthe
strings.Thelengthofthisdescriptionofsisthesumof
1. Thelengthoftheprogram InterpretLanguage ,whichwecantaketobetheconstantc.
2. ThelengthofPwhichbydefinitionisK2(s).
Thisprovesthedesiredupperbound.

Historyandcontext

[edit]

AlgorithmicinformationtheoryistheareaofcomputersciencethatstudiesKolmogorovcomplexityand
othercomplexitymeasuresonstrings(orotherdatastructures).
TheconceptandtheoryofKolmogorovComplexityisbasedonacrucialtheoremfirstdiscoveredbyRay
Solomonoff,whopublisheditin1960,describingitin"APreliminaryReportonaGeneralTheoryof
InductiveInference"[3]aspartofhisinventionofalgorithmicprobability.Hegaveamorecomplete
descriptioninhis1964publications,"AFormalTheoryofInductiveInference,"Part1andPart2
inInformationandControl.[4][5]
AndreyKolmogorovlaterindependentlypublishedthistheoreminProblemsInform.Transmission[6]in
1965.GregoryChaitinalsopresentsthistheoreminJ.ACMChaitin'spaperwassubmittedOctober1966
andrevisedinDecember1968,andcitesbothSolomonoff'sandKolmogorov'spapers.[7]
Thetheoremsaysthat,amongalgorithmsthatdecodestringsfromtheirdescriptions(codes),thereexists
anoptimalone.Thisalgorithm,forallstrings,allowscodesasshortasallowedbyanyotheralgorithmup
toanadditiveconstantthatdependsonthealgorithms,butnotonthestringsthemselves.Solomonoffused
thisalgorithm,andthecodelengthsitallows,todefinea"universalprobability"ofastringonwhich
inductiveinferenceofthesubsequentdigitsofthestringcanbebased.Kolmogorovusedthistheoremto
defineseveralfunctionsofstrings,includingcomplexity,randomness,andinformation.
WhenKolmogorovbecameawareofSolomonoff'swork,heacknowledgedSolomonoff'spriority.[8]For
severalyears,Solomonoff'sworkwasbetterknownintheSovietUnionthanintheWesternWorld.The
generalconsensusinthescientificcommunity,however,wastoassociatethistypeofcomplexitywith
Kolmogorov,whowasconcernedwithrandomnessofasequence,whileAlgorithmicProbabilitybecame
associatedwithSolomonoff,whofocusedonpredictionusinghisinventionoftheuniversalpriorprobability
distribution.Thebroaderareaencompassingdescriptionalcomplexityandprobabilityisoftencalled
Kolmogorovcomplexity.ThecomputerscientistMingLiconsidersthisanexampleoftheMattheweffect:"...
toeveryonewhohasmorewillbegiven..."[9]
data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color%3A

4/5

5/21/2015

KolmogorovcomplexityWikipedia,thefreeencyclopedia

ThereareseveralothervariantsofKolmogorovcomplexityoralgorithmicinformation.Themostwidely
usedoneisbasedonselfdelimitingprograms,andismainlyduetoLeonidLevin(1974).
AnaxiomaticapproachtoKolmogorovcomplexitybasedonBlumaxioms(Blum1967)wasintroducedby
MarkBurgininthepaperpresentedforpublicationbyAndreyKolmogorov(Burgin1982).

Basicresults

[edit]

Inthefollowingdiscussion,letK(s)bethecomplexityofthestrings.
Itisnothardtoseethattheminimaldescriptionofastringcannotbetoomuchlargerthanthestringitself
theprogram GenerateFixedString abovethatoutputssisafixedamountlargerthans.
Theorem:Thereisaconstantcsuchthat
s.K(s)|s|+c.

UncomputabilityofKolmogorovcomplexity

[edit]

Theorem:ThereexiststringsofarbitrarilylargeKolmogorovcomplexity.Formally:foreachn,thereis
astringswithK(s)n.[note1]
Proof:Otherwiseallinfinitelymanypossiblestringscouldbegeneratedbythefinitelymany[note2]programs
withacomplexitybelownbits.
Theorem:Kisnotacomputablefunction.Inotherwords,thereisnoprogramwhichtakesastringsas
inputandproducestheintegerK(s)asoutput.
ThefollowingindirectproofusesasimplePascallikelanguagetodenoteprogramsforsakeofproof
simplicityassumeitsdescription(i.e.aninterpreter)tohavealengthof1 400 000bits.Assumefor
contradictionthereisaprogram
functionKolmogorovComplexity(strings)

data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color%3A

5/5