Entropy, joint entropy, mutual information

Attribution Non-Commercial (BY-NC)

33 views

Entropy, joint entropy, mutual information

Attribution Non-Commercial (BY-NC)

- Nadeem
- Site Layout
- 7C's of Business Communication
- Information Theory
- 113118033_dp.pdf
- information Theory
- Kolmogorov Complexity & Shannon Entropy
- dcngisEncyclopidia
- 1995 - Cortez, Menezes De Sequeira, Pereira - Classification and coding of contour information
- geçici
- Specific Competence Indicators
- SG
- Triple Cross Algorithms
- Exercise 15.4
- McCowan et al 1999(1).pdf
- Council of Europe Language Levels
- Lecture 11
- information theory- shriram
- Critical Review Essay
- The Functions of Visual Management

You are on page 1of 14

Information theory deals with the problem of efficient and reliable transmission of information It specifically encompasses theoretical and applied aspects of

- coding, communications & communications networks - complexity and cryptography - detection and estimation - learning, Shannon theory, and stochastic processes

Satellite communications: Reed Solomon Codes (also CD-Player) Viterbi Algorithm Public Key Cryptosystems Compression Algorithms Huffman Lempel-Ziv MP3 JPEG MPEG Modem Design with Coded Modulation Codes for Recording ( CD, DVD )

Information is knowledge that can be used i.e. data is not necessarily information

We:

1) specify a set of messages of interest to a receiver 2) and select a message to be transmitted 3) sender and receiver build a pair

Information in its most restricted technical sense is an ordered sequence of symbols that record or transmit a message. It can be recorded as signs, or conveyed as signals by waves. Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Information theory is a branch of science that deals with the analysis of a communications system

The higher the likelihood of a particular outcome, the less information that outcome conveys Consider tossing a coin If the coin is biased such that it lands with heads up 99% of the time, there is not much information conveyed when we flip the coin and it lands on heads. Information

I(X j )

log

1 P( X j )

log

1 Pj

log Pj

1. log2 units are bits (from 'binary') 2. log3 units are trits(from 'trinary') 3. loge units are nats (from 'natural logarithm') 4. log10 units are Hartleys

ENTROPY

Entropy is a measure of the uncertainty associated with a random variable.

The term usually refers to the Shannon entropy, which quantifies the expected value of the information contained in a message, usually in units such as bits. The concept was introduced by Claude E. Shannon in his 1948 paper A Mathematical Theory of Communication. Shannon's entropy represents an absolute limit on the best possible lossless compression of any communication.

Entropy is the measurement of the average uncertainty of information The entropy of a message is thus a measure of how much information it really contains.

H entropy p probability X random variable with a discrete set of possible outcomes (X0, X1, X2, Xn-1) where n is the total number of possibilities

n 1 n 1

Entropy H ( X )

j 0

p( x) j log p( x) j

j 0

p( x) j log

1 p ( x) j

8

The entropy of X can also be interpreted as the expected value of log 1/p(x), where X is drawn according to probability- mass function p(x). Thus H(X) = Ep log 1/p(X)

Properties of Entropy

1. Entropy is always positive. H(X) 0 since 0 p(x) 1 for all p(x). 2. We can change bases freely. Hb(X) = (logba) Ha(x) since logbp = logba logap

Q2.

10

JOINT ENTROPY

12

CONDITIONAL ENTROPY

The conditional entropy H(Y|X) is

13

CHAIN RULE

The entropy of a pair of random variable is the entropy of one plus the conditional entropy of the other.

Also

- Site LayoutUploaded byWilliam Bonney
- 7C's of Business CommunicationUploaded byatif_khan8877
- Information TheoryUploaded byamit mahajan
- information TheoryUploaded byrahulsingle
- NadeemUploaded byshahimulk
- 113118033_dp.pdfUploaded byFaisal Ichal
- Kolmogorov Complexity & Shannon EntropyUploaded byflaneur_Sg8231
- dcngisEncyclopidiaUploaded byVidhya Packirisamy
- 1995 - Cortez, Menezes De Sequeira, Pereira - Classification and coding of contour informationUploaded byManuel Menezes de Sequeira
- geçiciUploaded bySıtkı Akkaya
- Specific Competence IndicatorsUploaded byViorica Toma
- SGUploaded byKapil_Kaushik_8847
- Triple Cross AlgorithmsUploaded byEmmett Boedeker
- Exercise 15.4Uploaded byLeonard Gonzalo Saavedra Astopilco
- McCowan et al 1999(1).pdfUploaded byMargarita E. García Moscoso
- Council of Europe Language LevelsUploaded byIan Christie
- Lecture 11Uploaded byOjuolape Gbolahan
- information theory- shriramUploaded byapi-269884528
- Critical Review EssayUploaded byezryn9291
- The Functions of Visual ManagementUploaded byAndreSantana
- Detection of Historical Period in Symbolic Music TextUploaded byCoie8t
- CrcUploaded byrachna009
- my five sensesUploaded byapi-384073421
- Multilevel Pixel Colour Values Scrambling Based on Shifting Image TechniqueUploaded byATS
- DynUploaded byFranko Milovan
- no 5Uploaded byMeylinda Dwi Anjarwati
- My-Family-16-May-2019-545Uploaded byJulia Gálvez Álvarez
- Entropy and Information in Science and Political EconomyUploaded byBuran Won
- Shannon Information and Kolmogorov ComplexityUploaded byDaniel Hardt Lima
- Q.no1 December 2016 MHA 04Uploaded byrakesh kumar

- 6 Wind LoadsUploaded byJovan Bucol
- SAP_ASE_Installation_Guide_Linux_en.pdfUploaded bySahitram Kumar
- Evidence My Holiday TripUploaded byLiliana Monroy
- Psych 3 Summer 2018 Syllabus-1Uploaded bySharlene Whitaker
- Test9y10Uploaded bySantiago De los Santos
- Hospital Information SystemUploaded bysumdkb1
- PppUploaded byImrul Huda Ayon
- Science Horizon November 2011Uploaded byWaiLinLi
- ENGR112 HW2Uploaded byRenan Santos
- Impact of 13 Valent Pneumococcal Conjugate Vaccine.20Uploaded bysakabuana
- CAU BANH XICH 40 TAN.pdfUploaded byduongpn
- CSR Trends and Realities in Romania - EY_ Romania & CSRmedia.ro_eNUploaded byCSRMedia Network
- Flight Performance of a Package Delivery Quad Rotor BiplaneUploaded byBreno Brito Miranda
- Dam Safety Management - StematiuUploaded byRamonaCruceru
- Map of Père Lachaise CemeteryUploaded byDemetra Vg
- Robert Dreyfuss - Wikipedia.pdfUploaded byAmr Mabrouk
- p424yUploaded byRandy Velez
- Are Transformation Tables Needed to Create Transformation AttributesUploaded byjani
- Likes and Dislikes - TeenagersUploaded byRosanna Bellaubi
- 1.the Effectivness of English Communication Using Pearson Global English Curriculum Ijariie5576Uploaded byKiran Soni
- A tulpaUploaded bySandip Jambhule
- Biopharma Manufacturing TechnologyUploaded bysushaantb400
- Arthur Anderson ToolbookUploaded byRupee Rudolf Lucy Ha
- Numeric Water Level Indicator_www.projectspediaUploaded byNiketan Mishra
- love and capitalismUploaded bysymphonynanu
- SMU MB0050 assignment answerUploaded byAnkur Sharma
- p3Uploaded byAmsalu Setey
- Night MailUploaded byGamini Jayasinghe
- GHOST by Connie Jay MaraanUploaded byAssumpta Minette Burgos
- DefectUploaded byAlaa Hosney