0 Up votes0 Down votes

3 views4 pagesMay 08, 2012

© Attribution Non-Commercial (BY-NC)

PDF, TXT or read online from Scribd

Attribution Non-Commercial (BY-NC)

3 views

Attribution Non-Commercial (BY-NC)

- NLP and Entropy
- Learning From Their Mistakes 011
- Section 6.5 Page 349 to 355
- Bayesian Networks in Mastermind (2003)
- Course Schedule
- On the Smarandache Pseudo-number Sequences
- section6
- Command Comparison of the Conversion Operations Between STEP7 Micro and STEP5 Www.otomasyonegitimi.com
- Lakshmanan, Hariharan
- Easy qb
- Filter Pap Suct Me as Demons Trt n
- Master Thesis Dorfinger
- 2006 Annual Report
- Binary Numbers
- Jerry
- soalanpptp1
- Sample Cpp e
- Cryptographic Applications
- Number System & Program Design
- 1 yl

You are on page 1of 4

This is a closed-book quiz, except that three 8.5 11 sheets of notes are allowed. There are three problems on the quiz of approximately equal value. The problems are not necessarily in order of diculty. A correct answer does not guarantee full credit and a wrong answer does not guarantee loss of credit. You should concisely indicate your reasoning and show all relevant work. The grade on each problem is based on our judgment of your level of understanding as reected by what you have written. If we cant read it, we cant grade it. If you dont understand a problem, please ask. Calculators are allowed, but probably wont be useful

Cite as: Robert Gallager, course materials for 6.450 Principles of Digital Communications I, Fall 2006. MIT OpenCourseWare (http://ocw.mit.edu/), Massachusetts Institute of Technology. Downloaded on [DD Month YYYY].

Problem Q1.1 (35 points) Consider a quaternary-valued source governed by a four-state Markov model that is op erating in its steady state.

As shown in Figure 1, when in a state i, a transition to state [(i + 1) mod 4] occurs with

probabity p, where p is a parameter, and such a transition causes a source symbol [(i + 1)

mod 4] to be produced; when no transition occurs, the source symbol i is produced.

(a) Suppose the memory in the source is ignored and a simple Human code is used to code each symbol. What average rate R1 in bits per source symbol is the source compressed to? (b) Determine the average rate R in bits per source symbol achievable by the best code for this source if the memory is taken into account. Express your answer in terms of the binary entropy function HB () = log2 () (1 ) log2 (1 ). Is there a value of p for which your answer is close to that of part (a)? Explain. (c) Suppose the source is partitioned into consecutive length-2 blocks, inter-block statisti cal dependencies are ignored, and a 2-to-variable length code of mininum expected length is used, which exploits the intra-block dependencies. The achievable average rate in bits per source symbol in this case is guaranteed to be at least as low as R2 = aHB (b) + c, where HB () is again the binary entropy function. Determine a, b, and c, expressing your answers in terms of the parameter p.

1;1p 2;1p

2;p

1;p

3;p

0 0;1p

0;p 3 3;1p

Cite as: Robert Gallager, course materials for 6.450 Principles of Digital Communications I, Fall 2006. MIT OpenCourseWare (http://ocw.mit.edu/), Massachusetts Institute of Technology. Downloaded on [DD Month YYYY].

Problem Q1.2 (35 points) Consider encoding the set of positive integers and assume that a probability distribution on these integers satises pi > pj for i < j. The table below gives the binary representa tion, the unary code and the unary-binary code for the rst few positive integers. (Recall that the unary-binary code is used in LZ77. This code works by viewing an integer j as a pair of integers (n, m) where n = log j + 1 and m = j 2n1 . Then j is mapped into the unary code for n concatenated with the n 1 bit binary representation for m.) j 1 2 3 4 5 6 binary rep. unary code unary-binary code 1 10 11 100 101 110 1 01 001 0001 00001 000001 1 010 011 00100 00101 00110

(a) Does the binary representation constitute a prex-free code? Explain. How about the unary and unary-binary codes? (b) Show that if an optimum (in the sense of minimum expected length) prex-free code is chosen for any given pmf (subject to the condition pi > pj for i < j), the code word lengths satisfy li lj for all i < j. Use this to show that for all j 1 lj log j + 1 (c) The asymptotic eciency of a prex-free code for the positive integers is dened to be limj log j . What is the asymptotic eciency of the unary-binary code? lj (d) Explain how to construct a prex-free code for the positive integers where the asymp totic eciency is 1. Hint: Replace the unary code for the integers n = log j + 1 in the unary-binary code with a code whose length grows more slowly with increasing n.

Cite as: Robert Gallager, course materials for 6.450 Principles of Digital Communications I, Fall 2006. MIT OpenCourseWare (http://ocw.mit.edu/), Massachusetts Institute of Technology. Downloaded on [DD Month YYYY].

Problem Q1.3 (True or False) 30 points For each of the following, state whether the statement is true or false and briey indicate your reasoning. No credit will be given without a reason. (a) Suppose X and Y are binary valued random variables with probability mass function given by pX (0) = 0.2, pX (1) = 0.8 pY (0) = 0.4 and pY (1) = 0.6. The joint probability mass function that maximizes the joint entropy H(X, Y ) is given by pX,Y (, ) X =0 X=1 Y =0 0.08 0.32 Y =1 0.12 0.48 (b) For a discrete memoriless source X with alphabet X = {1, 2, . . . , M }, let Lmin,1 , Lmin,2 , and Lmin,3 be the normalized average length in bits per source symbol for a Human code over X , X 2 and X 3 respectively. Then there exists a specic probability mass function 2 for source X for which Lmin,3 > 3 Lmin,2 + 1 Lmin,1 . 3 (c) Assume that a continuous valued rv Z has a probability density that is 0 except over the interval [A, +A]. Then the dierential entropy h(Z) is upper bounded by 1 + log2 A. Also h(Z) = 1 + log2 A if and only if Z is uniformly distributed between A and +A.

- NLP and EntropyUploaded bymyoxygen
- Learning From Their Mistakes 011Uploaded byJoseph Hayes
- Section 6.5 Page 349 to 355Uploaded byKnoll711
- Bayesian Networks in Mastermind (2003)Uploaded byFranck Dernoncourt
- Course ScheduleUploaded byno_one1410
- On the Smarandache Pseudo-number SequencesUploaded byDon Hass
- section6Uploaded byCynthia Solomon
- Command Comparison of the Conversion Operations Between STEP7 Micro and STEP5 Www.otomasyonegitimi.comUploaded bywww.otomasyonegitimi.com
- Lakshmanan, HariharanUploaded byVenkatesh Nenavath
- Filter Pap Suct Me as Demons Trt nUploaded bySallam Mohammed
- Master Thesis DorfingerUploaded byJawad Bashir
- 2006 Annual ReportUploaded byDmitry Grenishen
- Easy qbUploaded byPol Asuncion
- Binary NumbersUploaded byMoyses Moy
- JerryUploaded byElchin Hasanalizade
- soalanpptp1Uploaded byEta Puzi
- Sample Cpp eUploaded byJJ
- Cryptographic ApplicationsUploaded bylkijhbgags
- Number System & Program DesignUploaded byKhaled Hasan Khan
- 1 ylUploaded byAsiff Mohammed
- 3.OPERATORS_MB.pptUploaded byPriyankar Kumar
- Handouts 1 - MathematicsUploaded byAngela Aprina Kartika Putri
- Qualifying Round 2013 AnswersUploaded byRegnvaldr Alev
- Conversion of Binary to Hexadecimal and Octal SystemUploaded byRamón Rivas
- 600Computer MCQUploaded byAli Rehman
- Feature ANalysis Technology Accessing STatistics (In a Corpus): Technical Report v1.5Uploaded byash jackk
- ELEMENTS the PSYCHOPHYSICS-02-English-Gustav Theodor FechnerUploaded bygabriel brias buendia
- poTypes.docxUploaded byRohit Agarwal
- Variable Table PCS Rev3Uploaded byguigazedo
- 08 21090Uploaded bygeopic

- 196040ff93bae9c9df8453dd90baad58c99544feUploaded byapi-127299018
- 702a75fe02ff62e5ae5b74646d18da512a724d02Uploaded byapi-127299018
- 84cc54402d20f47cfb17a201658f04d3414edd9bUploaded byapi-127299018
- fd35a5c22ca4c8787de76166eded73a103ad02afUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- f7895136c515d2e469f0a12e5c6c54190d8b4be1Uploaded byapi-127299018
- 23b8563760685fae18fea95b388f930edafaa7deUploaded byapi-127299018
- b151d72e4b9391412234cd9302dd07e679cc861aUploaded byapi-127299018
- 125c5da1e1b8536345a1c6a9876c51b5653bed34Uploaded byapi-127299018
- 187e9e5268ed74af09b10b60a622384cec2efa74Uploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018
- UntitledUploaded byapi-127299018

- Sample Cover Letters 4Uploaded byalkanzy
- Hamdi Small ChangeUploaded byshumispace319
- 12 Dozen Places to Educate Yourself Online for FreeUploaded byKashif Ameer
- Journal of Interdisciplinary History Volume 22 Issue 1 1991 [Doi 10.2307%2F204587] Review by- Patricia Herlihy -- Politics and Public Health in Revolutionary Russia, 1890-1918by John F. Hutchinson;HeaUploaded byJoaquim Moreno
- Startup - Startup_Guide.pdfUploaded byEdu Bianchi
- 2017-6 Submarine Concept Design BrochureUploaded byndesigngmail
- In Medis Res, Spring 2005Uploaded byMIT Comparative Media Studies/Writing
- MIT15 769F10 Lec01 Operations StrategyUploaded byrakeshblr
- MIT Career Development Handbook Resume and Cover Letter.pdfUploaded bydis4sites
- 2002 Estrin Reconfig OriginsUploaded byRamakrishnan Swaminathan Thirumangalam
- SOP Robotics MITUploaded bySoumya Chatterjee
- Architecture after pakistanUploaded byNawaz Ali F
- CV of Jacie YangUploaded byJacie Yang
- Bathtub DynamicsUploaded byspithridates
- Handout 3 MITUploaded byRoger Mario Lopez
- MIT Sloan Desmond LimUploaded byDesmond Lim Huang Hui
- Master of Science in Management Studies - M.I.TUploaded byMarvin971
- carta de interesesUploaded byMohit Karnani
- Does Gender MatterUploaded byMoeshfieq Williams
- Recommendation Use of MIT in Cosmetics 12 12 13Uploaded bypgdermatologos
- Steven Holl - Simmons HallUploaded byElla Vargas
- Pennsbury's Top Scholars 2013Uploaded byBucksLocalNews.com
- Adewunmi Akolade General CVUploaded bymofiyinfolu
- rethinking learning in the digital ageUploaded byapi-454708004
- Annals of Improbable Research vol15no4Uploaded bymarxbert
- MIT Special AppUploaded byKrupa Saraiya
- Core ModelUploaded byHarinder Kaur Naina
- Bits & Bytes Vol.1 No1Uploaded byEmmanuel123250
- Emeging Ethical IssuesUploaded byAsmawi Mohamad
- India PrespectUploaded byAmber Agrawal

## Much more than documents.

Discover everything Scribd has to offer, including books and audiobooks from major publishers.

Cancel anytime.