5 views

Uploaded by gowshiselva

This is a presentation done by me as partial requirement to the module, Information Theory and Coding. This was made based on the paper published by Claude Shannon.

- Stocastic Rainfall Model
- 2010-09-18 22.21.06.610 Formal.Assessment (Initial).WinSAT
- A Systematic Procedure for the Design of a Cold Rolling Mill
- Stochastic Processes by Joseph T Chang
- Fall 2012 Seasonal Catalog
- Ds 50 Brochure
- Electrical and Electronics Engineering
- Signals and Systems
- Continuous Review Perishable Inventory Systems Models and Heuristics
- Novel Approaches to Improve the Particle Size Distribution Prediction of a Classical Emulsion Polymerization Model
- Generador de Tonos Mod-300MB
- Stochastic Processes and Markov Chains
- MIT6_003S10_lec02
- 2a.MA41_Model_1pdf
- dictaat.pdf
- ID3 Presentation
- A Coherent View on Entropy
- 9
- final6711F07
- Grey Markov Theory

You are on page 1of 20

Claude shannon

Gowshigan.S

Asian Institute of Technology Thailand

Information Theory and Coding 20000471

07.04.2014

Gowshigan.S (AIT)

Presenation

07.04.2014

1 / 20

Overview

1

Introduction

Communication system

The discrete noiseless channel

The discrete source of Information

Graphical Representation of Markov Process

Ergodic and Mixed sources

Choice, Uncertainty And Entropy

The Entropy of An Information Source

The Discrete channel with Noise

Second Section

Gowshigan.S (AIT)

Presenation

07.04.2014

2 / 20

Introduction

Claude Elwood Shannon was an American mathematician, electronic engineer, and cryptographer known as the father of information theory.

Shannon is famous for having founded information theory with a landmark

paper that he published in 1948

Gowshigan.S (AIT)

Presenation

07.04.2014

3 / 20

Communication System

Gowshigan.S (AIT)

Presenation

07.04.2014

4 / 20

communicated to the receiving terminal.

A sequence of letters as in a telegraph of Teletype system

A single function of time f(t ) as in radio or telephony

A function of time and other variables as in black and white television

Two or more functions of time, say f(t ), g(t ), h (t )

Several functions of several variables

Various combinations also occur, for example in television with an

associated audio channel

suitable for tranmssion over the channel. Ex Vocoder systems ,

Televisions, Frequency modulation

Gowshigan.S (AIT)

Presenation

07.04.2014

5 / 20

to receiver

transmitter , reconstruction the message from the signal

is intended

Gowshigan.S (AIT)

Presenation

07.04.2014

6 / 20

increase linearly with time

The capacity to transmit information can be specified by giving the

rate of increase

How much information source to be described mathematically?

How much information bits per second in produced ?

Frequent letters: shortest sequence

Infrequent Letters: Longest sequence

Gowshigan.S (AIT)

Presenation

07.04.2014

7 / 20

Gowshigan.S (AIT)

Presenation

07.04.2014

8 / 20

1

produces symbols according to certain probabilities is known as

Stochastic Process

sequence of symbols from a finite set may be considered a discrete

source

Gowshigan.S (AIT)

Presenation

07.04.2014

9 / 20

Consider the Example (B):

Using the same five letters let the probabilities be .4, .1, .2, .2, .1,

respectively, with successive choices independent. A typical message

from this source is then:

AAACDCBDCEAADADACEDA

E A D C A B E D A D D C E C A A A A A D.

Gowshigan.S (AIT)

Presenation

07.04.2014

10 / 20

Gowshigan.S (AIT)

Presenation

07.04.2014

11 / 20

Among the possible discrete Markov Process there is a group with

special properties of significance in communication theory and this

special class is called ergodic process and corresponding sources are

called ergodic sources

In ergodic process every sequences produced by process is the same is

statical properties and that is the one distinguish this from other

process

Gowshigan.S (AIT)

Presenation

07.04.2014

12 / 20

Lets define a quantity which will measure how much information is produced by a process

Suppose we have a set of possible events whose probabilities of occurrence are p1,p2 ,..... pn.

and

If there is such a measure, say H( p1,p2,,,,pn )

H should be continuous in the Pi

If all the pi are equal, Pi=(1/n) then H should be a monotonic

increasing function of n. With equally likely events there is more

choice, or uncertainty, when there are more possible events.

Gowshigan.S (AIT)

Presenation

07.04.2014

13 / 20

should be the weighted sum of the individual values of H. The

meaning of this is illustrated in Fig. 6. At the left we have three

Gowshigan.S (AIT)

Presenation

07.04.2014

14 / 20

T he only H

the three above assumptions is the form

Psatisfying

n

H = K

P

log

P

i

i

1

w here K is a positive constant, merely amounts to choice of unit measure

T he entropy in the case of two probabilities p and q = 1 p , namely

H = (p log p + q log q) is plotted as a function of p

Gowshigan.S (AIT)

Presenation

07.04.2014

15 / 20

previous graph has some interesting properties

1

H = 0 if and only if all the pi but one are zero, this one having the

value unity. Thus only when we are certain of the outcome does H

vanish. Otherwise H is positive.

For a given n, H is a maximum and equal to log n when all the pi are

equal (i.e., 1/n ). This is also intuitively the most uncertain

situation.

possibilities for the first and n for the second. Letp(i, j) be the

probability of the joint occurrence of i for the first and j for the

second. The entropy of the joint

is

Pevent

n

H(x, y ) = i,j p(i, j) log p(i, j)

and it is easily shown that H(x, y ) 6 H(x) + H(y )

4

p1 , p2 , , , , , , pn increasesH

Gowshigan.S (AIT)

Presenation

07.04.2014

16 / 20

C onsider a discrete source of the finite state type considered above. For

each possible state i there will be a set of probabilities pi (j) of producing

the various possible symbols j.

Thus there is an entropy Hi for each state.

The entropy of the source will be defined as the average of these Hi

weighted in accordance withPthe probability

P of occurrence of the states in

question: H = i Pi Hi = i,j Pi pi (j) log pi (j)

This is the entropy of the source per symbol of text.

Gowshigan.S (AIT)

Presenation

07.04.2014

17 / 20

Discussion

produces a sequence of letters chosen from among A, B, C , D with

probabilities 21 , 14 , 81 , 18 successive symbols being chosen independently.We

have H = ( 12 log 21 + 14 log 14 + 28 log 18

= 74 bitspersymbol

Gowshigan.S (AIT)

Presenation

07.04.2014

18 / 20

transmitted signal and received signal

P,i (, j)

C = Max (H(x) Hy (y ))

Gowshigan.S (AIT)

Presenation

07.04.2014

19 / 20

The End

Gowshigan.S (AIT)

Presenation

07.04.2014

20 / 20

- Stocastic Rainfall ModelUploaded bySudharsananPRS
- 2010-09-18 22.21.06.610 Formal.Assessment (Initial).WinSATUploaded bydanypam
- A Systematic Procedure for the Design of a Cold Rolling MillUploaded byNatalie Jennings
- Stochastic Processes by Joseph T ChangUploaded byMainak Ghosh
- Fall 2012 Seasonal CatalogUploaded byPrinceton University Press
- Ds 50 BrochureUploaded byRaulMesa
- Electrical and Electronics EngineeringUploaded byFernandoGoganzález
- Signals and SystemsUploaded byMohammad Gulam Ahamad
- Continuous Review Perishable Inventory Systems Models and HeuristicsUploaded byasd
- Novel Approaches to Improve the Particle Size Distribution Prediction of a Classical Emulsion Polymerization ModelUploaded byscoobardmembrane
- Generador de Tonos Mod-300MBUploaded byTOM
- Stochastic Processes and Markov ChainsUploaded byGra Vity
- MIT6_003S10_lec02Uploaded byAmandeep Singh
- 2a.MA41_Model_1pdfUploaded byNikhilPrakash
- dictaat.pdfUploaded byAlex Helder Cordeiro Oliveira
- ID3 PresentationUploaded byMihir Shah
- A Coherent View on EntropyUploaded byforizsl
- 9Uploaded byPronadeep Bora
- final6711F07Uploaded bySongya Pan
- Grey Markov TheoryUploaded byAjeng Alfia Fadilla Teguhputri
- Preface 2001 Measurement and Instrumentation Principles Third EditionUploaded byMotasim_m
- hammingUploaded byterradeath
- Protocol ExUploaded bySandeepReddyGoluguri
- 12-1 midtermUploaded byapi-308984218
- IJCSI-10-5-1-71-77Uploaded bynikhilvbhatt9591
- 216_vgGray R. - Probability, Random Processes, And Ergodic PropertiesUploaded bydavid
- 18 Earthquake PredictionUploaded byjafeelnm
- Logics Dynamics Based Upon {(Measurable, Driven), (Wake Up, Speed Up), (Custom, Event), (Handle, Hold)}, Which Settles Switch FashionUploaded bysfofoby
- Common Customization Cores of Sensitive Sensor Effects Using {(Measurable, Driven), (Wake Up, Speed Up), (Custom, Event), (Handle, Hold)} Symbolic Switching FashionUploaded bysfofoby
- 1-363 Simulation of City BusUploaded byanggihadiwijaya123

- Data structures and algorithms to support interactive spatial analysis using dynamic Voronoi diagUploaded byLuz Nidia Gómez Luna
- Direct Inward DialUploaded bySantiago Ladrón De Guevara Villar
- 6_16v6i1_1Uploaded byMayur Swamy
- Bio 130 Chapter 25 Hand OutUploaded byhkalloli@gmail.com
- CCNA Dis3 - Chapter 5 - Routing With a Distance Vector Protocol_ppt [Compatibility Mode]Uploaded byhttp://heiserz.com/
- Form Penilaian Full Paper 138Uploaded byAngelaMonalisaKurniawan
- 54630_SPE-PAPERUploaded byRichars Gonzalez
- chapter 4 of homework 2 databasesUploaded bySaurabh Kapadia
- Storage Tank (API 650) _ Home Welding & InspectionUploaded byHussain Ali
- Ds 006405Uploaded byCristian Flores Martínez
- Phase-2 Voided SlabUploaded byKetan Dhameliya
- Heron TrianglesUploaded byhumejias
- bp1799.pdfUploaded byAhmed Bahei
- FAARFILD Airport Pavement DesignUploaded bywirya
- GX Works2 BrochureUploaded byAlvaro Villalba Torres
- 7_substitution_postUploaded byapi-3767370
- Common EmitterUploaded byHarsha
- How CAESAR is Calculating NEMA23 Moments About Resoltn Point - Intergraph CADWorx & AnalysisUploaded byTon Phichit
- Gibbs Energy Modeling of Binary and Ternary Molten Nitrate Salt SUploaded byCavanStone
- Enumeration of Bacteria in Milk Samples and Presumptive Test for ColiformsUploaded byAlda Yana
- Gas Laws Packet 2 ANSWERSUploaded byJoseph Turner
- Stress Strain CurveUploaded byDianaCruz
- Avramopoulos-2012-Architectural Innovation A Game-theoretic Approach-000.pdfUploaded bybobster124u
- jemh104Uploaded byahadaziz
- EpiCyclic gear trainUploaded byRaghavMaheshwari
- Prem Chowdhry CVUploaded byPrem Chowdhry
- EGA324-C1-Surname-StudentID-000000(17-18)Uploaded byijaz afzal
- Airlines NDT Conf Digital-RadiographyUploaded byRamakrishnan AmbiSubbiah
- Inferred e Bp Pid ListUploaded byWilson Bueno
- 175 Alternators - KTRv3Uploaded byQuintin Du Plooy