## Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

Information Theory and Source Coding

$

• Scope of Information Theory 1. Determine the irreducible limit below which a signal cannot be compressed. 2. Deduce the ultimate transmission rate for reliable communication over a noisy channel. 3. Deﬁne Channel Capacity - the intrinsic ability of a channel to convey information. The basic setup in Information Theory has: – a source, – a channel and – destination. The output from source is conveyed through the channel and received at the destination. The source is a random variable S, & %

. · · · . the choice of present symbol does not depend on the previous choices. sk−1 } with probabilities P (S = sk ) = pk where k = 0. s2 . • Information: Information is deﬁned as & % . 1.' which takes symbols from a ﬁnite alphabet i. Source is memoryless i.e.e. · · · .. Source generates symbols that are statistically independent. 2. s1 . S = {s0 . k − 1 and k −1 $ pk = 1 k=0 The following assumptions are made about the source 1. 2.

' $ I (sk ) = logbase 1 pk • In digital commmunication. the ‘base ’ is always 2. & % . data is binary.

e. sun rising in the east as opposed to that of the occurence of the Tsunami a Give & % . 2.e.. f or pk = 1a .. Information is always positive i. Information conveyed by a deterministic event is nothing i. I (sk ) ≥ 0 f or pk ≤ 1 examples about deterministic events. I (sk ) = 0.' $ Properties of Information 1.

More information is conveyed by a less probable event than a more probable event I (sk ) > I (si ). I (sk ) is never negative $ 4.e.' 3.. Information is never lost i. f or pk < pi a program that runs without faults as opposed to a program that gives a segmentation fault aa & % .

then its entropy is bounded by 0 ≤ H(s) ≤ log2 K & % .' Properties of Information $ • Entropy: The Entropy(H(s)) of a source is deﬁned as: the average information generated by a discrete memoryless source. If K is the number of symbols generated by a source. Entropy deﬁnes the uncertainity in the source. K −1 H (s ) = k=0 p k I (s k ) • Properties of Entropy: 1.

. all other probabilities should be zero i. i = k .e. pi = 0. ∀ i. then H(s) = log2 K $ • Proof: – H (s ) ≥ 0 If the source has one symbol sk which has P (s = sk ) = pk = 1. If the source generates all the symbols equiprobably i.e. Therefore.' 2. ∀ k . If the source has an inevitable symbol sk for which pk = 1 then H (s ) = 0 3. 1 pk = K .. from the deﬁnition of entropy & % . then according to the properties.

· · · . p1 .' K −1 $ H (s ) = k=0 pk log 1 pk 1 pk = pk log = 1 log(1) =0 Therefore H(s) ≥ 0 – H(s) ≤ log2 K Consider a discrete memoryless source which has two diﬀerent probability distributions & p0 . pk % .

· · · . we consider the following inequality. · · · . loge x ≤ x − 1 & (2) % . s1 . sk We begin with the equation K −1 $ pk log k=0 qk pk 1 K −1 = pk loge loge 2 k=0 qk pk (1) To analyze Eq.' and q0 . q1 . qk on the alphabet S = s0 . 1.

% . k=0 qk log 1 qk = log2 K So. K −1 $ qk pk 1 K −1 pk ≤ loge 2 k=0 pk log k=0 qk −1 pk K −1 ≤ 1 loge 2 1 K −1 qk − k=0 k=0 pk (3) ≤ K −1 (1 − 1) = 0 loge 2 =⇒ k=0 pk log qk pk ≤0 K −1 Suppose.2 in Eq. 1. qk = & 1 K.' Now. using Eq.∀ k then.

3 becomes K −1 K −1 $ pk log pk ≤ 0 k=0 K −1 K −1 pk log qk − k=0 =⇒ − k=0 K −1 pk log K ≤ k=0 K −1 pk log pk 1 pk 1 pk =⇒ k=0 pk log K ≥ k=0 K −1 K −1 pk log =⇒ log K k=0 K −1 pk ≥ k=0 pk log =⇒ k=0 pk log 1 pk ≤ log2 K Therefore H(s) ≤ log2 K & % .' in Eq.

H(s) has a maximum value when p0 = 1 2 as demonstrated in Figure 1. & % . Let the probabilities of symbols 0 and 1 be p0 and 1 − p0 respectively. 1}.' $ Entropy of a binary memoryless channel Consider a discrete memoryless binary source shown deﬁned on the alphabet S = {0. The entropy of this channel is given by H(s) = −p0 log2 p0 − (1 − p0 ) log2 (1 − p0 ) According to the properties of Entropy.

' $ H(s) 1 0 1/2 1 p Figure 1: Entropy of a discrete memoryless binary source & % .

- assignment_3.pdf
- The Bead Store sells material for customers to make their own jewelry.
- Continuous Decoding of Turbo
- Good Meetings Part 1
- ch4
- Data Communication Basic
- Best Books for Gate in Cse
- IJAIEM-2013-05-28-070
- Bsc Sp Seminar Re
- resnote11
- Binary Phase Shift Keying Modulation Based Balanced Power Adaptation Transmission of Images Over Wireless Channels
- Time Table Ch2
- CBIS_week3_handout_2002
- Course Review of Social Network Analysis (coursera)
- ch08
- Binary Search
- Approximation Algorithms for NP-Hard Problems
- Stats Tables All
- unit01-coordinategrids
- Barriers to Effective Communication and How to Overcome Them by Arun K. Behera & Bijay K. Tripathy
- Factoring Power Point
- bzcat
- Psyclone
- ch05n
- Topological Methodology
- Management Information System
- CDMA Network Coverage Optimization in South-Eastern Nigeria
- 1 Introduction
- Potwd 15 Na Rs 04 s
- How Does One Gets Job at Google

- Blind Channel Shortening for MIMO-OFDM System Using Zero Padding and Eigen Decomposition Approach
- Noise Analysis on Indoor Power Line Communication Channel
- Bit Error Rate Performance of MIMO Spatial Multiplexing with MPSK Modulation over Rayleigh Fading Channel
- A New Transmission Scheme for MIMO â€“ OFDM
- Adaptive Resource Allocation in MIMO-OFDM Communication System
- Efficient Channel Assignment Schemes In 802.11 WLAN

- UT Dallas Syllabus for se3306.001 06f taught by Gopal Gupta (gupta)
- UT Dallas Syllabus for te3307.001.08f taught by Charles Bernardin (cpb021000)
- UT Dallas Syllabus for cs5333.501 05f taught by Timothy Farage (tfarage)
- UT Dallas Syllabus for ecs2305.003.10f taught by Charles Bernardin (cpb021000)
- UT Dallas Syllabus for cs3305.501 05f taught by Timothy Farage (tfarage)
- UT Dallas Syllabus for cs5333.001.09f taught by Jorge Cobb (jcobb)
- UT Dallas Syllabus for cs5333.001 06f taught by Nancy Van Ness (nancyvn)
- UT Dallas Syllabus for cs2305.501.06f taught by Timothy Farage (tfarage)
- UT Dallas Syllabus for cs5333.501 06s taught by Nancy Van Ness (nancyvn)
- UT Dallas Syllabus for cs2305.001 05s taught by (wxw021000)
- UT Dallas Syllabus for math5320.501 05f taught by Thomas Butts (tbutts)
- tmp1204.tmp
- UT Dallas Syllabus for te3307.001 05f taught by Charles Bernardin (cpb021000)
- UT Dallas Syllabus for ecs2305.003.10f taught by Charles Bernardin (cpb021000)
- UT Dallas Syllabus for mthe5320.501.09f taught by Titu Andreescu (txa051000)
- UT Dallas Syllabus for math5320.501.07f taught by Thomas Butts (tbutts)
- UT Dallas Syllabus for mthe5324.501.11f taught by Titu Andreescu (txa051000)

Sign up to vote on this title

UsefulNot usefulRead Free for 30 Days

Cancel anytime.

Close Dialog## Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

Close Dialog## This title now requires a credit

Use one of your book credits to continue reading from where you left off, or restart the preview.

Loading