EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)

Academic Course Description

SRM University
Faculty of Engineering and Technology
Department of Electronics and Communication Engineering
EC1028 ELEMENTS OF INFORMATION THEORY AND CODING
Seventh Semester, 2016-17 (Odd semester)

Course (catalog) description

This course introduces the basic principles of encoding, error detection, and error correction, decoding,
mutual information, and channel capacity, which will be extremely useful in understanding the working
of a digital communication system.

Instructional Objectives (IOs):

1. To analyze the process of coding for analog and discrete sources and the mathematical model
for information sources.
2. To solve problems on error detection and error correction for various types of codes.
3. To understand the principles of Huffman codes and to solve problems therein.
4. To study the properties of Entropy and the principles of Shannon-Fano coding.
5. To learn the concepts of mutual information, channel capacity, and Shannon’s Main Theorem.

Compulsory/Elective course: Compulsory.

Credit hours: 3 credits.

Course coordinator: Mrs. R. Dayana, A.P (O.G)/ECE

Instructor(s)

Email
Name of the Batch-Strength, Office Office
@ktr.srmuniv.ac. Consultations
instructor Room Slots location phone
in
1 Dr.V.Nithya B1-58 [Room No: 12.30 to 1.30
TP12S7 - nithya.v
AP (Sr.G)/ECE TP 1305] pm
2 Dr.M.Sangeetha B1-58 [Room No: 12.30 to 1.30
TP9S5 2055 sangeetha.m
AP (Sr.G)/ECE TP 1306] pm
3 Mrs. R. Dayana, B1-58 [Room No: 12.30 to 1.30
TP1006A 2061 dayana.r
AP (OG)/ECE TP 1401] pm
4 Dr. Sabitha Gouni B1- 58 [Room No: 12.30 to 1.30
TP11S - sabitha.g
AP (Sr.G)/ECE TP 1402] pm
5 Mr.K.Paldurai B1- 58 [Room No: 12.30 to 1.30
TP1206A - paldurai.k
AP (OG)/ECE TP 1502] pm
6 Ms. S. Ranjani, B2-57 [Room No: 12.30 to 1.30
TP13S4 7869 ranjani.s
AP/ITCE TP 1404] pm
7 Ms. N. Showme, B2-57 [Room No: 12.30 to 1.30
TP13S4 7869 showme.n
AP(OG)/ ITCE TP 1403] pm
Page 1 of 8
 EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)
Email
Name of the Batch-Strength, Office Office
@ktr.srmuniv.ac. Consultations
instructor Room Slots location phone
in
8 Ms.C.Vimala,
B2-57 [Room No: TP13S5/ 12.30 to 1.30
AP(OG)/ ITCE 2153 vimala.c
TP 1503] TP1306A pm
9 Mrs.J.Subhashini B2-57 [Room No: 12.30 to 1.30
TP12S6 2069 subhashini.j
AP (Sr.G)/ECE TP 1501] pm
10 Mrs.P.Malarvezhi B2-57 [Room No: 12.30 to 1.30
TP1203A 2064 malarvizhi.p
AP (OG)/ECE TP 1406] pm
11 Ms.D.Vijayalaksh
B2-57 [Room No: 12.30 to 1.30
mi TP1203A 2064 vijayalakshmi.d
TP 1405] pm
AP (OG)/ECE

Relationship to other courses

Pre-requisites : MA1024
Assumed knowledge :
Following courses :

Syllabus Contents

UNIT I-SOURCE CODING (9 hours)

Model of signaling system - Mathematical models for information sources – Encoding a source alphabet
– ASCII code – Radix r code – Miscellaneous codes - A Logarithmic measure of information – Coding for
Discrete sources – Coding for analog sources (Optimum quantization) – Coding techniques for analog
sources.
UNIT II-ERROR DETECTING AND ERROR CORRECTING CODES (10 hours)
Simple parity checks – CRC codes – Hamming weight – Hamming distance – Minimum distance decoding
– Single / Double parity checks – Hamming codes – Linear block codes – Cyclic codes – Syndrome
calculation – Block encoders and Decoders.
UNIT III-VARIABLE-LENGTH CODES – HUFFMAN CODES (10 hours)
Unique decoding – Instantaneous codes and its construction – The Kraft’s inequality – Shortened block
codes – The McMillan’s Inequality – Huffman codes and its special cases – Extensions of a code –
Huffman codes Radix r – Noise in Huffman coding probabilities – Use of Huffman codes – Hamming
Huffman coding
UNIT IV-ENTROPY AND SHANNON’S FIRST THEOREM (5 hours)
Entropy and its Mathematical properties – Entropy and coding – Shannon-Fano coding – Entropy of a
Markov process – The Adjoint system – Robustness of Entropy.
UNIT V-MUTUAL INFORMATION, CHANNEL CAPACITY & SHANNON’S MAIN THEOREM (11 hours)
Information channel – Capacity of a Binary symmetric channel – System entropies – Mutual information
– Definition of channel capacity – Uniform channel – Conditional mutual information – Random
encoding - Average random code – Fano bound – Converse of Shannon’s theorem.

TEXT BOOKS
1. Hamming, Richard W, “Coding and Information Theory”, Prentice Hall Inc., NJ, 1986.
2. Proakis J. G., “Digital Communications”, McGraw Hill Inc., 4th Edition, NY, 2001.

Computer usage (if any): Nil

Page 2 of 8
 EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)
Professional component
General - 0%
Basic Sciences - 0%
Engineering sciences & Technical arts - 100%
Professional subject - 0%

Broad area: Communication | Signal Processing | Electronics | VLSI | Embedded

Test Schedule

S. No. Test Tentative Date Portions Duration

1 Cycle Test-1 01.08.2016 1 to 11 Session -
2 Cycle Test-2 01.09.2016 12 to 28 Session -
Session 1 to 40 15
3 Surprise test - minutes
(MCQ Pattern)

4 Model Test 20.10.2016 Session 1 to 45 3 Hrs

All sessions / 3 Hrs

5 University Exam 11.11.2016
Units

Mapping of Instructional Objectives with Program Outcome

This course introduces the basic principles of encoding, error detection, and Correlates
to program
error correction, decoding, mutual information, and channel capacity,
outcome
which will be extremely useful in understanding the working of a digital
communication system. H M L
1. To analyze the process of coding for analog and discrete sources
and the mathematical model for information sources. a e

2. To solve problems on error detection and error correction for

various types of codes. a e
3. To understand the principles of Huffman codes and to solve
problems therein. a e

4. To study the properties of Entropy and the principles of Shannon-

a e
Fano coding.
5. To learn the concepts of mutual information, channel capacity,
a e
and Shannon’s Main Theorem.

H: high correlation, M: medium correlation, L: low correlation

Draft Lecture Schedule

Problem
Text /
Session Topics solving
Chapter
(Yes/No)
UNIT I-SOURCE CODING
1 Model of signaling system No [1] chapter - 1

Page 3 of 8
 EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)
Problem
Text /
Session Topics solving
Chapter
(Yes/No)
Mathematical models for information sources, Encoding a No
2 source alphabet
3 ASCII code, Radix r code No
4 Miscellaneous codes No [1] chapter - 5
5 A Logarithmic measure of information Yes
6 Coding for Discrete sources Yes
[2] chapter - 3
7 Coding for analog sources (Optimum quantization) No
8 Coding techniques for analog sources No
UNIT II-ERROR DETECTING AND ERROR CORRECTING CODES
9 Simple parity checks Yes
10 CRC codes Yes
Hamming codes , Hamming weight, Hamming distance, Yes [1] chapter - 2
11 Minimum distance decoding

12 Single / Double parity checks Yes

13 Linear block codes Yes
14 Cyclic codes Yes
15 Syndrome calculation Yes [2] chapter - 8
16 Block encoders and Decoders No
UNIT III-VARIABLE-LENGTH CODES – HUFFMAN CODES
17 Unique decoding Yes

18 Instantaneous codes and its construction Yes

21 The Kraft’s inequality Yes

22 Shortened block codes Yes
23 The McMillan’s Inequality No
Yes
24 Huffman codes and its special cases [1] chapter – 4
Yes
25 Extensions of a code
Huffman codes Radix r Yes
26
Yes
27 Noise in Huffman coding probabilities
Use of Huffman codes ,Hamming Huffman coding Yes
28
UNIT IV-ENTROPY AND SHANNON’S FIRST THEOREM
29 Entropy and its Mathematical properties Yes
30 Entropy and coding Yes
[1] chapter – 6
31 Shannon-Fano coding Yes
32 Entropy of a Markov process Yes
33 The Adjoint system No
34 Robustness of Entropy Yes
UNIT V-MUTUAL INFORMATION, CHANNEL CAPACITY & SHANNON’S MAIN THEOREM
35 Information channel Yes [1] chapter –
36 Capacity of a Binary symmetric channel Yes 7,8,10
37 System entropies Yes
Page 4 of 8
 EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)
Problem
Text /
Session Topics solving
Chapter
(Yes/No)
38 Mutual information Yes
39 Definition of channel capacity No
40 Uniform channel No
41 Conditional mutual information Yes
42 Random encoding Yes
43 Average random code No
44 Fano bound No
45 Converse of Shannon’s theorem. No

Teaching Strategies
The teaching in this course aims at establishing a good fundamental understanding of the areas
covered using:
 Formal face-to-face lectures
 Tutorials, which allow for exercises in problem solving and allow time for students to resolve
problems in understanding of lecture material.
Evaluation Strategies
Cycle Test – I - 10%
Cycle Test – II - 10%
Model Test - 20%
Surprise Test - 5%
Attendance - 5%
Final exam - 50%

Prepared by: Mrs. R. Dayana, Assistant Professor (Ordinary Grade), Department of ECE

Dated: 29-06-2016

Revised by: - Revision No.: - Date of revision: -

Course Coordinator Admin. Coordinator Professor In-Charge HOD

(Mrs. R. Dayana) (Mrs. N. Saraswathi) (Dr. B. Ramachandran) (Dr.T. Rama Rao)

Page 5 of 8
 EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)

Addendum

ABET Outcomes expected of graduates of B.Tech / ECE / program by the time that they graduate:
(a) An ability to apply knowledge of mathematics, science, and engineering

(b) An ability to design and conduct experiments, as well as to analyze and interpret data

(c) An ability to design a system, component, or process to meet desired needs within realistic
constraints such as economic, environmental, social, political, ethical, health and safety,
manufacturability, and sustainability.

(d) An ability to function on multidisciplinary teams

(e) An ability to identify, formulate, and solve engineering problems

(f) An understanding of professional and ethical responsibility

(g) An ability to communicate effectively

(h) The broad education necessary to understand the impact of engineering solutions in a
global, economic, environmental, and societal context

(i) A recognition of the need for, and an ability to engage in life-long learning

(j) A knowledge of contemporary issues

(k)An ability to use the techniques, skills, and modern engineering tools necessary for
engineering practice.

Program Educational Objectives

1. Graduates will perform as a successful professional engineer in related fields of Electronics and
Communication Engineering.
2. Graduates will pursue higher education and/or engage themselves in continuous professional
development to meet global standards.
3. Graduates will work as a team in diverse fields and gradually move into leadership positions.
4. Graduates will understand current professional issues, apply latest technologies and come out with
innovative solutions for the betterment of the nation and society.

Page 6 of 8
 EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)

Course Teachers Class Signature

Page 7 of 8
 EC1028 ELEMENTS OF INFORMATION THEORY AND CODING (JUNE 2016)

Course Coordinator Academic Coordinators Professor In-Charge HOD

(Mrs. V.Padmajothi) (Dr.Diwakar.R. Marur) (Dr.P. Aruna Priya) (Dr.S.Malarvizhi)
(Mrs.J.Subhashini)

Page 8 of 8