Information Theory and Coding Full Book

INTRODUCTION TO
INFORMATION THEORY
AND CODING
(Electronics & Communication Engineering/Telecommunication Engineering)
Authors
Prof. SUNIL S HARAKANNANAVAR,B.E., M. Tech., (Ph.D.,)

S. G. Balekundri Institute of Technology,
Belagavi-590010.
Prof. Veena I Puranikmath, B.E., M. Tech., (Ph.D.,)

S. G. Balekundri Institute of Technology,
Belagavi-590010.
Dr. Prashanth C R, B.E., M. Tech., Ph.D.

Dr. Ambedkar Institute of Technology,
Bangalore-560056.
BookName : Introduction to Information Theory and Coding
Author : Prof. Sunil S Harakannanavar, Prof. Veena I Puranikmath

and Dr. Prashanth C R
CopyRight :Prof. Sunil S Harakannanavar

Assistant Professor
Department of Electronics and Communication Engineering
S. G.Balekundri Institute of Technology
Shivabasavnagar, Belagavi-590010.
First Edition : July 2018
Pages : 1 to 000
All rights reserved. No part of this publication can be reproduced in any

form by any means without the prior written permission from the publisher.
All the contents, data, information, views opinions, charts, tables, figures,
graphs etc., that are published in this book are the sole responsibility of the
authors. Neither the publishers nor the editors in anyway are responsible for the
same.
ISBN: 978-93-87756-20-5
Published by
MayasPublication®
# 45/5, Unathur& Post.Attur Tk.,
Salem Dt. Tamilnadu, India – 636112
www.eijfmr.commaya1984kannan@gmail.com
editoreijfmr@gmail.com
+91 9655933100
ii
Preface
Information Theory and Coding may be the fastest growing technologies in the field
of Communication and network security. One of the ramifications of that growth is a
dramatic increase in the number of professions-where an understanding of these
technologies is essential for success-and a proportionate increase in the number and
types of students taking course to learn about them.
Features of the Book

Several features of this text are designed to make it particularly easy for students to
understand information theory and coding.
Course Objectives
Course objectives are clearly defined at initial for all the chapters.
Structure
This textbook is designed for students with little or no background in
telecommunications or data communications. For this reason, we use a bottom-up
approach. With this approach, students learn first about information system (bottom
up approach) followed by generating the code words and code vectors using linear
block codes and cyclic codes respectively.
Visual Approach
The book presents highly technical subject matter without complex formulas by using
a balance of text and figures. More than 200 figures accompanying provide a visual
and intuitive opportunity for understanding the material. Figures are particularly
important and explaining coding concepts, which are based on connections and
transmissions. Both of these ideas are easy to grasp visually.
Highlighted Points
We emphasize important concepts in highlighted boxes for quick references and
immediate attention.
Recommended Reading
Each chapter includes a list of books and sites that can be used for further reading.
iii
Preface
Key Terms
Each chapter includes a list of key terms for the student.
Practice Set
Each chapter includes a practice set designed to reinforce and apply Salient concepts.
It consists of two parts: descriptive questions and model exam question papers (end
of all the chapters). Descriptive questions are intended to test the students first-level
understanding of the material presented in the chapter. Multiple choice questions are
designed to create motivation for the students to appear in the competitive
examinations. Examination questions require deeper understanding of the material.
Index
The book contains an extensive list of acronyms.
Organization
The following lists the course material in the organization of the book:
1. Chapter 1 discusses the brief Introduction of Information Theory.
2. Chapter 2 deals with the generation of code words Source coding.
3. Chapter3 describes the various Communication Channels in the field of
Communication.
4. Chapter 4 discusses Error Control Codes for Linear Block Codes.
5. Chapter 5 discuses Binary Cyclic Codes.
6. Chapter 6 discusses Important Cyclic codes
7. Chapter 7 deals with the convolution codes.
Instructional Materials
Instructional materials for both the student and the teacher are provided. The
solutions to exercises contain both the explanation and answer including full colored
figures or tables when needed. The PowerPoint presentations are more
comprehensive and include text and figures.
Contents
The book is divided into six parts. The first part is an introduction to Information
theory; the second part is Source Coding; the third part is Informational Channels; the
iv
Preface
fourth chapter is Error Control Coding; Binary Cyclic codes are discussed in Chapter
5; Chapter 6 covers some important Cyclic codes and Chapter 7 covers the
Convolution codes. The following summarizes the contents of each part.
Part one: Information Theory: Introduction, Measure of information, Information

content of message, Average Information content of symbols in Long Independent
sequences, Average Information content of symbols in Long dependent sequences,
Markov Statistical Model of Information Sources, Entropy and Information rate of
Markoff Sources.
RBT Level: (L1, L2, L3)
Part two: Source Coding: Source coding theorem, Prefix Codes, Kraft McMillan
Inequality property – KMI. Encoding of the Source Output, Shannon’s Encoding
Algorithm.Shannon Fano Encoding Algorithm, Huffman codes, Extended Huffman
coding, Arithmetic Coding, Lempel – Ziv Algorithm.
Part Three: Information Channels: Communication Channels. Channel Models,

Channel Matrix, Joint probability Matrix, Binary Symmetric Channel, System
Entropies, Mutual Information, Channel Capacity, Channel Capacity of: Binary
Symmetric Channel, Binary Erasure Channel, Muroga,s Theorem, Continuous
Channels.
Part Four: Error Control Coding: Introduction, Examples of Error control coding,
methods of Controlling Errors, Types of Errors, types of Codes, Linear Block Codes:
matrix description of Linear Block Codes, Error Detection and Error Correction
Capabilities of Linear Block Codes, Single Error Correcting hamming Codes, Table
lookup decoding using Standard Array.
Part Five: Binary Cyclic Codes: Binary Cyclic Codes: Algebraic Structure of Cyclic
Codes, Encoding using an (n-k) Bit Shift register, Syndrome Calculation, Error
Detection and Correction.
v
Preface
Part Six: Some important Cyclic codes Golay Codes, BCH Codes.
Part Seven:Convolution codes:Convolution Codes: Convolution Encoder, Time

domain approach, Transform domain approach, Code Tree, Trellis and State
Diagram, The Viterbi Algorithm.
How to Use the Book

This book is written for both an academic and a professional audience. The book can
be used as a self-study guide for interested professionals. As a textbook, it can be used
for a one-semester course. The following are some guidelines.
 Parts one to three are strongly recommended.
 Parts four to seven can be covered to detect and correct the errors that lead to
improve the efficiency of communication system.
The book is intended for undergraduate and postgraduate students of Electronics

and Communication Engineering, Computer Science and Engineering, Information
Technology and Telecommunication Engineering. The course material in this book is
already taught to students for many years by the author; and the book is written with
better understanding of the needs of the students and the practicing engineers in the
subject.
Acknowledgements
It is obvious that the development of a book of this scope needs the support of many
people.
Peer Review
The most important contribution to the development of a book such as this comes
from peer reviews. We cannot express our gratitude in words to the many reviewers
who spent numerous hours reading the manuscript and providing us with helpful
comments and ideas. We would especially like to acknowledge the contribution of the
following reviewers for this book.
1. M.Pavithra, M.E.,
Assistant Professor, Tamilnadu.
vi
Preface
2. Rajendiran
Assistant Professor, Suriya Institute of Engineering and
Technology, Vilupuram, Tamilnadu.
3. Dr. Senthil Kumar
MGR University, Professor of Computer Science
Chennai.
Just like any other book, this book being the first edition is bound to have some
errors and omissions, which may please be bought to the notice of the authors or
publisher to come with improvement in the second edition. All suggestions and
Critical comments are most welcome.
We pledge our sincere sense of gratitude to our parents and all our family
members for their constant support and encouragement.
We wish to thank our respected his holiness Pujya Shri. Dr. Siddharama
Swamiji, President, SSET, Belagavi, Shri S. G. Sambaragimath, Chairman, S. G. B. I.
T, Belagavi, Dr. S. S. Salimath, Principal, S G Balekundri Institute of Technology,
Belagavi, Dr. K. S. Prabhushetty, Professor and Head, Department of Electronics and
communication Engineering, S. G. Balekundri Institute of Technology, Belagavi for
their encouragement to carry out this work.
Also we would like to express our sincere gratitude to Panchajanya Vidyapeetha
Welfare Trust, Bangalore, Dr. C Nanjundaswamy, Principal, Dr. Ambedkar Institute
of Technology, Bangalore and Dr. Yamunadevi C R, Professor and Head, Department
of Telecommunication Engineering, Dr. Ambedkar Institute of Technology, Bangalore
for their constant support and encouragement.
Our sincere thanks to Dr. Venugopal K. R., Vice-Chancellor, Bangalore
University, Bangalore and Dr. K. B. Raja, Professor and Chairman, University
Visvesvaraya College of Engineering, Bangalore for their constant support and
encouragement.
We wish to thank all our staff and teaching faculties for their inputs at various
stages of writing the book including the feedback from our students.
I express my grateful thanks to Dr. R. Mayakkanan and their members of the staff,
for their efforts in publishing this book. Our sincere thanks to Mr. Muthu S, page
designer for excellent typesetting of the manuscript.
Suggestions for improvement of the book are welcome.

vii
About the Authors
Prof. Sunil S Harakannanavar completed his Bachelor of Engineering

in the stream of Electronics and Communication Engineering from Sri
Taralabalu Jagadguru Institute of Technology, Ranebennur and
masters in the field of Microelectronics and Control Systems from
Nitte Mahalinga Adyanthaya Memorial Institute of Technology,
Nitte. Presently he is working as Assistant Professor with S. G.
Balekundri Institute of Technology Belagavi. He is pursuing his
Ph.D at Visvesvaraya Technological University, Belagavi and his area
of interests includes Computer Vision, Pattern Recognition and Biometrics. He is a life
member of Indian Society for Technical Education, New Delhi and IEAE.
Prof. Veena I. Puranikamth graduated in the field of Electronics and

Communication Engineering from S. G. Balekundri Institute of
Technology, Belagavi and completed masters in Digital
Communication and networking at Godutai Engineering College for
Women, Kalaburagi. Presently she is working as Assistant Professor
with S. G. Balekundri Institute of Technology Belagavi. She is
pursuing her Ph.D at Visvesvaraya Technological University,
Belagavi and her area of interest are wireless sensor networks and
communication systems.
Dr. Prashanth C R received the B.E degree in Electronics, ME degree

in Digital Communication and Ph.D degree from Bangalore
University, Bangalore. He is currently working as a Professor,
Department of Telecommunication Engineering, Dr. Ambedkar
Institute of Technology, Bangalore. His research interests include
Computer Vision, Pattern Recognition and Biometrics. He is a life
member of Indian Society for Technical Education, New Delhi,
Member of IEEE, IACSIT, ACM and Fellow of Institution of Engineers.
July 2018
AUTHORS
viii
Table of Contents
Preface
Contents Page. No
1. Information Theory
Learning Objectives
1.1 Introduction...........................................................................................................01
1.2 Basics of Information System..............................................................................01
1.3 Measure of Information..................................................................................... 03
1.3.1 Unit of Information................................................................................... 03
1.3.2 Properties of Information......................................................................... 04
1.3.3 Average Information Content................................................................. 09
1.3.4 Properties of Entropy................................................................................ 18
1.3.5 Extension of Zero memory Source.......................................................... 25
1.4 Entropy of source with memory........................................................................ 30
1.4.1 Markoff Stochastic Model..........................................................................31
1.4.2 Entropy and Information rate of Markoff Sources.............................. 31
Things to be remember
Descriptive Questions
Multiple Choice questions
References
2. Source Coding
Learning Objectives
2.1 Introduction..............................................................................................................57
2.2 Types of Codes....................................................................................................... 58
2.2.1 Block Codes...................................................................................................58
2.2.2 Non Singular Codes................................................................................... 58
2.2.3 Singular Codes............................................................................................ 58
2.2.4 Uniquely Decodable Codes....................................................................... 59
2.2.5 Instantaneous Codes.................................................................................. 59
ix
Table of Contents
2.2.6 Optimal Codes............................................................................................ 60

2.3 Prefix Codes........................................................................................................... 60
2.4 Kraft Mc-Millan Inequality.................................................................................. 62
2.5 Code Efficiency and Redundancy....................................................................... 64
2.6 Shannon’s Encoding Theorem............................................................................. 65
2.7 Shannon Fano Encoding Algorithm................................................................... 81
2.7.1 Shannon Fano Ternary Coding Algorithm........................................... 89
2.8 Compact Code: Huffman Minimum Redundancy Code................................ 92
2.9 Shannon Hartley Law and its Implications..................................................... 110
2.10 Arithmetic Coding............................................................................................. 112
2.10.1 Arithmetic Algorithm for decoding.................................................... 117
2.11 Lempel-Ziv Coding.......................................................................................... 119
2.12 Lempel-Ziv-Welsh Coding.............................................................................. 122
References
3. Information Channels
Learning Objectives
3.1 Introduction...............................................................................................................135
3.1.1 Representation of a Channel.......................................................................135
3.2 Joint Probability....................................................................................................... 138
3.2.1 Properties of Joint Probability Matrix....................................................... 139
3.3 Entropy Function and Equivocation..................................................................... 142
3.4 Rate of Information transmission over a discrete channel................................ 144
3.4.1 Capacity of a discrete memoryless channel............................................. 144
3.4.2 Channel efficiency and Redundancy........................................................ 144
3.5 Special Channels...................................................................................................... 145
3.5.1 Symmetric Channel...................................................................................... 145
3.5.2 Binary Symmetric Channel......................................................................... 152
3.5.3 Binary Erasure Channel.............................................................................. 160
3.5.4 Noiseless Channel........................................................................................ 163
x
Table of Contents
3.5.5 Deterministic Channel................................................................................. 165

3.5.6 Cascaded Channel........................................................................................ 167
3.6 Estimation of Channel Capacity by Muroga’s Method..................................... 173
3.7 Continuous Channels............................................................................................. 179
References
4. Error Control Coding

Learning Objectives
4.1 Introduction...............................................................................................................187
4.2 Examples of Error Control Coding...................................................................... 187
4.3 Methods of controlling errors............................................................................... 188
4.4 Types of errors........................................................................................................ 189
4.5 Types of Codes........................................................................................................ 189
4.6 Linear Block Codes................................................................................................. 189
4.7 Matrix Description of linear block codes............................................................ 190
4.8 Encoding circuit for (n, k) linear block codes..................................................... 192
4.9 Syndrome Calculation Circuit.............................................................................. 193
4.10 Distance Properties of linear block codes........................................................ 206
4.11 Single error correcting Hamming Codes......................................................... 213
4.12 Table look-up decoding using the standard array......................................... 218
References
5. Binary Cyclic Codes

Learning Objectives
5.1 Introduction............................................................................................................. 231
5.2 Algebraic structure of cyclic codes....................................................................... 231
5.3 Properties of cyclic codes....................................................................................... 234
xi
Table of Contents
5.4 Encoding using (n-k) bit shift registers................................................................ 240

5.5 Syndrome calculation............................................................................................. 248
5.6 Error detection and correction.............................................................................. 249
References
6. Some Important Cyclic Codes

Learning Objectives
6.1 Introduction...............................................................................................................259
6.2 Galois field................................................................................................................259
6.2.1 Galois field of 2n............................................................................................262
6.2.2 Representation of Polynomials...................................................................263
6.2.3 Irreducible Polynomials...............................................................................264
6.3 Bose-Chaudhuri-Hocquenqhem (BCH) codes....................................................268
6.3.1 Generator Polynomial..................................................................................268
6.3.2 Degree of BCH code.....................................................................................269
6.4 Reed Solomon codes................................................................................................271
6.4.1 Properties of Reed Solomon codes.............................................................271
6.4.2 Generator Polynomial..................................................................................274
6.5 Golay codes...............................................................................................................275
References
7. Convolution Codes
Learning Objectives
7.1 Introduction..............................................................................................................281
7.2 Some definitions.......................................................................................................282
7.2.1 Code rate........................................................................................................282
7.2.2 Constraint length..........................................................................................282
7.3 Analysis of convolution encoder using time domain approach........................282
7.4 Analysis of convolution encoder using matrix method ....................................288
xii
Table of Contents
7.5 Analysis of convolution encoder using transform domain approach..............290

7.6 Graphical Approaches.............................................................................................301
7.6.1 State diagram.................................................................................................301
7.6.2 Code tree........................................................................................................305
7.6.3 Trellis diagram..............................................................................................307
7.7 Viterbi algorithm for convolution decoding........................................................309
7.7.1 Some definitions............................................................................................310
References
Appendix A: Model Question Papers

Index:
References
xiii
Chapter 1
Information Theory
Syllabus: Introduction, Measure of information, Information content of message,

Average Information content of symbols in Long Independent sequences, Average
Information content of symbols in Long dependent sequences, Markov Statistical Model of
Information Sources, Entropy and Information rate of Markoff Sources.
Contents
1. Information Theory Page. No
Learning Objectives
1.1 Introduction…………………………………………………………………………………01
1.2 Basics of Information System……………………………………………………………...01
1.3 Measure of Information……………………………………………………………………03
1.3.1 Unit of Information………………………………………………………………….03
1.3.2 Properties of Information…………………………………………………………...04
1.3.3 Average Information Content………………………………………………………09
1.3.4 Properties of Entropy………………………………………………………………..18
1.3.5 Extension of Zero memory Source…………………………………………………25
1.4 Entropy of source with memory…………………………………………….....................30
1.4.1 Markoff Stochastic Model…………………………………………………………...31
1.4.2 Entropy and Information rate of Markoff Sources……………………………….31
References
1
Claude. Elwood. Shannon was an American Mathematician,
Electrical Engineer and Cryptographer known as the “Father
of Information Theory”
Born: 30-04-1916
Died: 24-02-2001
Nationality: American
Occupation: Mathematician
2
1 INFORMATION THEORY
Learning Objectives
After reading this chapter, you will be able to:

Understand the need of information content of a message and average information content of a
system.
Develop a mathematical model to measure the information content for zero memory source.
Investigates the information content of the message by means of source efficiency and
redundancy.
Understand the properties of entropy.
Develop a Markov model to measure the information content for a memory dependent source.
1.1 Introduction
In 1948, Claude Shannon published "A Mathematical Theory of Communication", an
article in the Bell System Technical Journal. This work focuses on the problem of how
best to encode the information a sender wants to transmit. In this fundamental work
he used tools in probability theory, developed by Norbert Wiener, which were in their
nascent stages of being applied to communication theory at that time. Shannon
developed information entropy as a measure for the uncertainty in a message while
essentially inventing the field of information theory. The word information refers to
message or intelligence in information theory. Here, the message may be in the form
of voltage, power, speech, picture message such as facsimile or television or music
message. The source which produces these messages is called information source.
1.2 Basics of Information System

Information System is defined as the message is generated from the information
source and transmitted towards receiver through transmission medium. The Fig. 1.1
shows the block diagram of an information system.
3
Introduction to Information Theory and Coding
Figure 1.1: Block diagram of an Information System
Information Source: Information sources can be classified into two sections Analog
information Sources such as a microphone actuated by speech, TV camera scanning a
scene that emits one or more continuous amplitude electrical signals with respect to
time. Discrete Information sources such as a teletype or the numerical output of a
computer consists of a sequence of discrete symbols. The analog information source
can be transferred into a discrete information source through sampling and
quantization process. Further it is characterized by source alphabet, symbol rate,
source alphabet probabilities and probabilistic dependence of symbols in a sequence.
With respect to Fig. 1.1, now let us assume that the information source is a discrete
source that emits discrete message symbols {S1, S2,……….,Sq} with probabilities of
occurrence given by {P1, P2,…….., Pq} respectively. Then the sum of all these
probabilities must be equal to ‘1’ and is given in equation (1.1).
P1+P2+…..+Pq = 1 or ∑
………………….1.1
Source Encoder: Now let us consider the string of source symbols emitted from the
source alphabet S = {S1, S2,….,Sq} occurring at a rate of ‘rs’ symbols/sec as an input to
the source encoder. The source encoder converts the symbol sequence into a binary
sequence of 0’s and 1’s by assigning code-words to the symbols in the input sequence.
Transmitter: It couples the input message signal to the channel. Here, the signal
processing operations are also performed which includes amplification, filtering and
modulation.
Channel: When binary symbols are transmitted over the channel, the effect of noise is
to convert some of the 0’s into 1’s and some of the 1’s to 0’s. The signals are then said
to be corrupted by noise which leads to reduce the performance of the system.
4
Information Theory
Decoder: The source decoder converts the binary output of the channel decoder into
symbol sequence. Therefore, the function of the decoder is to convert the corrupted
signals into symbol sequence.
Receiver: The function of the receiver is to identify the symbol sequence and match it
with the correct sequence.
1.3 Measure of Information

Suppose if you are planning a trip to Delhi, Rajasthan, Odisha, Mawsynram in rainy
season. To determine the weather, you telephone the weather bureau and receive
following forecasts.
1. It rained heavily in Mawsynarm
2. There was a heavy rain last night in Rajasthan
Although both the statements refers the occurrence of heavy rainfall, but the amount
of information conveyed is different. The first statement contains very less
information since the weather in Mawsynarm is rainy all the time (there is no much
surprise). The forecast of the rain in Rajasthan contains more information, since it is
not an event that occurs more often. The amount of information in a message only
depends on the uncertainty of the underlying event rather than its actual content.
Now we can formalize this concept in terms of probabilities as follows:
Let us consider the communication system which transmits messages or independent
sequence of symbols from source alphabet S={S1, S2,….,Sq} with probabilities
P={P1,P2,…..,Pq} respectively. Now choose symbol ‘Sk’ for transmission at any instant
of time with a probability equal to ‘Pk’. Then the amount of information or self
information of message ‘Sk’ is given in equation (1.2).
Amount of Information (Ik)= log2
………………….1.2
1.3.1 Unit of Information
Let us consider the equation (1.2)., which shows the definition of amount of
information.
5
(Ik)= =
………………….1.3
Here, if the base of the logarithm is ‘2’, then the units are called “BITS” (Binary
Units). If the base is ‘3’, then the units are Trinits or Ternary. If the base is ‘10’, then
the units are “HARTLEYS” or “DECITS”. If the base is ‘e’, then the units are
“NATS”. Finally if the base is ‘r’, then the units are called r-array units. The most
widely used unit of information is “BITS” which has base 2.
1.3.2 Properties of Information

Some of the important reasons to choose the logarithmic expression for measuring
information can be listed as follows:
i) If there is more uncertainty about the message, information is also carried

more.(Eg., rainfall in summer days is rarely occurring event so carries more
information).
ii) The information ‘Ik‟ conveyed by a message cannot be negative. It has to be at
least zero.
Ik 0
iii) If receiver knows the message being transmitted, the amount of information
carried is zero.
If Pk=1 then Ik=0
iv) If ‘I1’ is the information carried by message ‘S1’ and ‘I2’ is the information carried
by ‘S2’, then the amount of information carried due to S1 and S2 is
I (S1, S2) = I1 +I2
v) If there are M=2N equally likely messages, then amount of information carried by
each message will be of ‘N’ bits.
6
Information Theory
EXAMPLE 1.1 Prove that I1, 2 = I1 + I2 where ‘I1’ represents the information carried by
message‘m1’ and ‘I2’ is the information carried by message ‘m2’.
Solution: We have, the amount of information is given by,

Ik= log2
The individual information carried by both messages m1 and m2 are
I1= log2 and I2 = log2
Hence the information carried by ‘m1’ and ‘m2’ is
I1, 2= log2 = (Ik) = log2* +
I1, 2= {log2 + log2 }
Therefore
I1, 2 = I1 + I2
Hence Proved
EXAMPLE 1.2 Consider a binary system emitting two symbols {0, 1} with
probabilities 0.25 and 0.75 respectively. Find the total information conveyed by bit 0
and bit 1.
Solution: We have, Self information content of a symbol can be calculated using

equation 1.2,
Ik= log2 bits
Probability of the occurrence of ‘0’ at the source output is P(0)=0.25 and the
probability of the occurrence of ‘1’ at the source output is P(1)=0.75. Therefore,
7
I0= log2 = log2 = = 2 bits
Similarly,
I1= log2 = log2 = = 0.415 bits
Therefore, the total information conveyed is, I = I0 + I1 = (2+0.415)
I= 2.415 bits
EXAMPLE 1.3 If there are ‘M’ equally likely and independent messages, then prove
that amount of information carried by each message will be of Ik=N bits, where M=2N
and N is an integer.
Solution: Since all the messages are equally likely and independent, probability of
occurrence of each message will be ( ). We know that, the amount of information is
given by,
Ik= log2 bits
We know that
Pk= then Ik=log2M
Ik= log2 2N
where M=2N
Ik = N log2 2 = N
Therefore
Ik = N bits
8
Information Theory
EXAMPLE 1.4 A source is emitting two symbols S0 and S1 with the probabilities
and respectively. Find the self information of the symbols in (i) Bits (ii) Trinits (iii)
Decits (iv) Nats.
Solution: (i) For Bits, We have,

Ik= log2 bits
Is0= log2 = log2 ⁄

= 1 bits

= 2 bits
(i) For Trinits, We have
Ik= log3 trinits
Is0= log3 = log3 = 0.63 trinits

⁄

= 1.26 trinits
(ii) For Decits, We have
Ik= log10 decits

= 0.30 decits

= 0.60 decits
(iii) For Nats, We have,
Ik= loge nats
Is0= loge = loge ⁄

= 0.69 nats
Is1= loge = loge ⁄

= 1.38 nats
EXAMPLE 1.5 Suppose a fair coin is tossed by binary source which emits ‘1’ for
head and ‘0’ for tail.
(i) What is the information content of each output from the source?
(ii) Consider if the source is memory less, then what will be the source
information of an m-bit block from the source.
9
Solution: There are two outputs ‘1’ and ‘0’ from the source. Here, both are equally
likely and the probability of each output is,
P(0)= and P(1)=
(i) Hence, the information content of each output is,
Ik= loge where k=0, 1
Therefore
I= log2 2 = 1 bit
(ii) If m-bits are to be transmitted as one message block, then the probability of
such a message will be,
P(x)= X X X X…… X (m-times) = ( )
Hence the information content is,
I(x) = log2 = log2
I(x)= m bits
EXAMPLE 1.6 Comment on the information content of the following messages

(i) Tomorrow the Sun will sets from the west
(ii) The mobile will ring in the next one hour
(iii) It will snow in Mangalore this winter.
Solution: The Information Content of the messages for the above three statements
are listed below as follows
(i) Since it is sure that Sun always sets from west and the probability of occurrence of
the event is high. Hence, it carries less or negligible information. i.e., Ik=0
(ii) It predicts about phone ring in the time span of one hour without mentioning the
exact time. Hence it carries moderate information.
(iii) In the winter season snow fall in Mangalore is very rare. Hence the probability of
occurrence of this event is very rare. So it carries large amount of information.
10
Information Theory
1.3.3 Average Information Content (ENTROPY) of Symbols in Long

Independent Sequences
For a zero memory source the emission of the current symbols are not dependent on
the emissions of the previous symbols. Let us consider zero memory source which
generates independent sequence of symbols S={S1,S2,.……..Sq} with probabilities
P={P1,P2,…..,Pq} respectively. Now let us consider long message of length ‘L’ emitted
by the source. Then it contains
P1L number of symbols of ‘S1’
………
………
………
……...
PqL number of symbols of ‘Sq’
Now, the self information of ‘S1’ is given by

= log2 bits
It is noticed that each symbol of ‘S1’ conveys information of bits and such P1L
number of ‘S1’ symbols are present on an average in a length of ‘L’ symbols. So, the
total information conveyed by symbol ‘S1’ is
P1L log2 bits
The total information conveyed by symbol ‘S2’ is
P2L log2 bits
Similarly, the total information conveyed by symbol ‘S3’ is
P3L log2 bits

………
………
11
Total information conveyed by symbol ‘Sq’ is
PqL log2 bits
Therefore, the total self information conveyed by all these symbols is,
ITotal = P1L log2 + P2L log2 + P3L log2 +…………. + PqL log2
The average information conveyed by the source is denoted by its Entropy H(s),
which is given by
H(s) =
* +
H(s) =
* +
=
= log2 + log2 + log2 +…………. + Pq log2
Hence, we can write
∑ bits/symbol
………………….1.4
Equation (1.4) represents the expression for the average information content of a
source ‘S’. If the baud rate of the system is ‘rs’ symbols/second, then the average rate
of the information is given by
Rs=H(s) * rs
=∑ bits/symbol * rs symbols/second
=[∑ * rs ] bits/second
Therefore, we can write

Rs=H(s) * rs bits/second
………………….1.5
12
Information Theory
EXAMPLE 1.7 A zero memory or discrete memory less source emits one of the six
symbols in every 3ms. The symbol probabilities are {0.5, 0.4, 0.01, 0.02, 0.03, 0.04}.Find
the average information rate of the source.
Solution: The entropy or the average information content of the source is given by
∑ bits/symbol
Substituting the probabilities {0.5, 0.4, 0.01, 0.02, 0.03, 0.04} given in the above
equation, we get
H(s) = { + + + + + }
H(s) = {0.5+0.52+0.06+0.11+0.15+0.18}
Therefore,
H(s) = 1.52 bits/symbol
Average rate of information is given by
It is given that a symbol is emitted for every 3ms. Therefore,
rs = = 333 symbols/second
So, average rate of the information is
Rs =1.52 * 333
Therefore,
Rs =506.16 bits/second
13
EXAMPLE 1.8 A zero memory source emits one of the four symbols in every 1ms.
The symbol probabilities are , -. Find
(i) The average information content of the source in bits/symbol, trinits/symbol,

Hartley/symbol and nats/symbol.
(ii) Average information rate of the source.
Solution: The entropy or the average information content of the source is given by
∑ bits/symbol
Substituting the given probabilities , - in the above equation, we get
H(s) = { ( ⁄ )
+
( ⁄ )
+
( ⁄ )
+
( ⁄ )
}
H(s) = {0.52+0.43+0.5+0.5}
Therefore,
H(s) = { ( ⁄ )
+
( ⁄ )
+
( ⁄ )
+
( ⁄ )
}
H(s) = {0.33+0.27+0.31+0.31}
H(s) = 1.22 trinits/symbol
H(s) = { ( ⁄ )
+
( ⁄ )
+
( ⁄ )
+
( ⁄ )
}
H(s) = {0.15+0.12+0.15+0.15}
H(s) = 0.57 Hartley/symbol
H(s) = { ( ⁄ )
+
( ⁄ )
+
( ⁄ )
+
( ⁄ )
}
14
Information Theory
H(s) = {0.36+0.29+0.34+0.34}
H(s) = 1.33 nats/symbol
ii) Average rate of information is given by

It is given that a symbol is emitted for every 1ms. Therefore,
Rs =1.95 * 1000
Therefore,
Rs =1950 bits/second
EXAMPLE 1.9 Suppose if ‘s’ represents the response of a single roll of a fair die.
Calculate the entropy of ‘s’.
Solution: We know that, a die is having a six faces. Hence, the probability of getting
any number is . i.e., Pi = . So, entropy of ‘s’ is given by
∑ bits/symbol
= }
bits/symbol
15
EXAMPLE 2.0 The output of information source consists of 100 symbols, 24 of which
occur with the probability of and the remaining 76 occur with a probability . If
the source emits 2000 symbols/sec, find the average information rate of the source.
Solution: The source entropy is given by

∑ bits/symbol
={ }
( ⁄ ) ( ⁄ )
= {0.11+0.04}
bits/symbol
Average rate of information is given by

It is given that source emits 2000symbol/second.
Therefore,
Rs =0.15 * 2000
Therefore,
Rs =300 bits/second
16
Information Theory
EXAMPLE 2.1A code is composed of dots and dashes. Assume that a dash is 3times
as long as a dot and has one-third the probability of occurrence. Find
(i) The average information content in a dot and dash
(ii) The entropy of dot - dash code
(iii) The average rate of the information, if a dot lasts for 10ms and this time is
allowed between symbols.
Solution: As we know that only dots and dashes are present we must have
Pdot +Pdash=1
………………….1.6
It is given that Pdash = Pdot
………………….1.7
Substituting equation (1.7) in equation (1.6) we get
Pdot + Pdot =1
Pdot =
………………….1.8
Substituting equation (1.8) in equation (1.7) we obtain
Pdash =
………………….1.9
(i) Information in a dot
Idot = = = 0.415 bits
Information in a dash is given by

Idash = = = 2 bits
(ii) The entropy of dot-dash code is
= { }
( ⁄) ( ⁄ )
17
}
Therefore,
bits/symbol
(iii) Since 3 dots and 1 dash with a dot lasting for 10ms, a dash for 30ms and with
10ms gap between two successful symbols, a total of 100ms is required to
transmit 4 symbols.
Therefore, symbol rate
rs= = 40 symbols/second
Rs =0.8113 * 40
Therefore,
Rs =32.45 bits/second
EXAMPLE 2.2 Find relationship between Hartleys, nats and bits.
Solution: From the definitions of Hartleys, nats and bits we have
I= Hartleys
………………….1.10
I= nats
………………….1.11
I= bits
………………….1.12
18
Information Theory
From equation (1.10),
1 Hartley = =
( ⁄ ) ( ⁄ )
= nats
= nats where =
Therefore,
1 Hartley = nats
Or,
1 Hartley =2.303 nats

Similarly,
1 Hartley = bits
We know that =
1 Hartley = bits
Or,
1 Hartley =3.32 bits
We have 1 nat = bits

But., =
1 nat = bits Or, 1 nat =1.443 bits
EXAMPLE 2.3 Calculate the entropy of a source in nats/symbol that emits one out of
four symbols P, Q, R, and S with probabilities , , and respectively.
19
Solution: The entropy of the source is given by

∑ bits/symbol
={ + }
={ }
( ⁄ ) ( ⁄ ) ( ⁄ ) ( ⁄ )
={0.52+0.5+0.5+0.33}
Therefore,
bits/second
………………….1.13
We know that
1nat=1.443 bits
1bit= nat = 0.693 nats
………………….1.14
Now, by substituting equation (1.14) in equation (1.13) we get
nats/symbol
nats/symbol
1.3.4 Properties of Entropy

Some of the interesting facts of entropy are discussed in this section.
1) Continuous property
The entropy function ∑ bits/symbol is continuous for
every independent variable ‘Pk’ in the interval (0, 1) and it vanishes at both Pk
=0 and Pk=1
i.e., H(s) = 0.
2) Symmetrical Property
The entropy function is a symmetrical function of its arguments
H [Pk , (1- Pk)]= H [(1- Pk), Pk] for all k=1, 2, 3……….., q
20
Information Theory
Ex: If PX = {P1, P2, P3} ; Py = { P2, P3, P1} ; Pz = { P3, P1, P2} then
H (Sx) = H (Sy) = H (Sz)

This means that, the value of H(S) remains same with the irrespective locations
of the probabilities.
3) Extremal Property
Consider the source with ‘q’ symbols S ={S1, S2, .....,Sq } with probabilities
P={P1, P2,....., Pq}. Then the entropy of ‘S’ is given by
∑ bits/symbol
We know that sum of all the probabilities is equal to unity.
P1+P2+.....+ Pq = ∑ =1 ………………….1.15
Since, the entropy H(S) has an upper bound which is proved by considering a
quantity [ ]. Therefore
= [∑ ] ∑ where ∑ =1
=∑ * +
=∑ Where =
∑ * + Where
Therefore,
………………….1.16
Figure 1.2: Graph of straight line y=x-1 and logarithmic curve y= ln x
21
From above graph it is noticed that the straight line Y=X-1 always lies above
the logarithmic curve Y= except at X=1. Thus, the straight line forms the
tangent to the curve at X=1. Therefore,
Multiplying by -1 on both sides we get

OR ≥
………………….1.17
If X= , then
Multiplying equation (1.17) both sides by ‘ ’then take summation for all
i=1,2,3, ………,q
∑ ∑ * +
Where
[ ]
………………….1.18
Multiplying equation (1.18) by we get
∑ *∑ ∑ +
………………….1.19
Where [∑ ∑ ]= 1-1=0
From equation (1.16) it is noticed that, LHS of equation (1.19) is
and RHS of equation (1.19) is always zero. Therefore
OR
Note: Suppose if then for all i=1, 2, ……..,q when this

condition satisfies then
H(s)max= bits/symbol
………………….1.20
22
Information Theory
Equation (1.20) is used, when the emitted symbols of the zero memory sources
are equiprobable.
4) Additive Property:
Suppose if the symbol ‘Sq’ is splitted into n sub symbols such that Sq = {Sq1,
Sq2, ....., Sqn}
With probabilities Pq= {Pq1, Pq2, .....,Pqn} then the entropy H(s) of splitted
symbols is greater than or equal to H(s).
H(s) ≥ H(s)
………………….1.21
This means that partitioning of symbols into sub symbols cannot decrease
entropy.
5) Source Efficiency: It is defined as the ratio of the average information
conveyed by the source to that of the maximum average information
= * 100%
………………….1.22
6) Source Redundancy: It is defined as
R =1- =* += * 100%
………………….1.23
EXAMPLE 2.4 The Source has generated twelve symbols in blocks of six at a rate of
1000 blocks/second. The first symbol in each block is always synchronization (same),
the remaining five are filled by any of the twelve symbols with equal probability.
Calculate the entropy rate of this source.
Solution: Given data q=12 symbols and rs = 1000 blocks/second
1st 2nd 3rd 4th 5th 6th
Sync
23
Since the symbols are generated in blocks of six, the total entropy HT is given by
HT= H1+H2+H3+H4+H5+H6
………………….1.24
In a given problem, 1st
bit is always same, which indicates no information is carried
by sure event (probability of the sure event is P1=1)
H1= P1
H1=0
The second, third, fourth, fifth and sixth position is occupied by any of the twelve
symbols with equal probability. Hence,
H2= H3= H4= H5= H6= H(S)max = =
H2= H3= H4= H5= H6=3.5 bits/symbol
Now, Substitute the values of H2, H3, H4, H5 and H6 in equation (1.24) we get
HT= {0+3.5+3.5+3.5+3.5+3.5}
HT=17.5 bits/symbol
Therefore, the entropy rate is given by

Rs = HT * rs bits/second
Rs = *
Rs = 17500 bits/second
24
Information Theory
EXAMPLE 2.5 A zero memory Source ‘S’ emits four symbols A, B, C and D with
probabilities 0.25, 0.15, 0.40 and 0.2 respectively. Find the efficiency and its
redundancy of the source.
Solution: The entropy of the source is given by
∑ bits/symbol
={ + + + }
={ + + +0.2 }
bits/symbol
To find maximum entropy we have
H(s)max = =
H(s)max =2 bits/symbol
The source efficiency is given by

= = = 0.945*100
=94.5%
The source redundancy is given by

R =1- =1- 0.945 = 0.055
R =5.5%
25
EXAMPLE 2.6 In a facsimile transmission of picture there are about 2 x

106pixels/frame. It is necessary to have eight brightness levels during good
reproduction. Assume that all the levels are equally likely to each other. Calculate the
rate of information if one picture is transmitted for every 4 seconds. Also, find the
source efficiency of this facsimile transmitter.
Solution: Total number of pixels in one frame = 2 x 106 pixels/ frame

Number of brightness levels = 8
Therefore, Total number of different frames possible = ( )
Since all the values are equally likely to occur, all these frames are equiprobable. So,
( )
H(s) = H(s)max = =
=2 x 106
H(s)max = 6 x 106 picture /symbol
It is said that one picture is transmitted for every 4 seconds. Therefore, the rate of
transmission is given by
rs= ( ) = 0.25 picture/second
Now, the average rate of information is obtained by
Rs = H(S) * rs bits/second
Rs =6 x 106 * 0.25
Rs = 15 x 106 bits/second
Source Efficiency is given by
= = =1
26
Information Theory
1.3.5 Extension of Zero Memory Source

In some coding techniques it is necessary to apply extension of zero memory sources.
To understand the concept of source extension, consider a binary source emitting
symbols S1 and S2 with their probabilities P1 and P2 respectively. Such that
P1+ P2 =1
………………….1.25
Then, the entropy for binary source is given by
∑ bits/symbol
………………….1.26
Now, 2nd extension of the binary source will be
[ ]
i.e., 22 =4 number of symbols is given by,

S1 S2 occurring with probabilities P1 P1=
S1 S2 occurring with probabilities P1 P2= P1 P2
S2 S1 occurring with probabilities P2 P1= P2 P1
S2 S2 occurring with probabilities P2 P2=
The entropy of 2nd extension of the binary source of given by
27
But we have,
=2 [ ]
From equation (1.26)
Similarly, the third extension of binary source will be 23 = 8 symbols and its entropy is
In general, the nth extension of binary source will be 2n symbols and its entropy is
………………….1.27
EXAMPLE 2.7 Let us consider the zero memory source S= {S0, S1, S3, S4} with their
probabilities {0.2, 0.6, 0.1, 0.1}. Find the entropy of source and the entropy of second
order extension.
Solution: The entropy of source is given by
∑ bits/symbol
={ + + }
Therefore,
bits/symbol
The entropy of the second order extension is given by
= 2 x 1.57
28
Information Theory
Therefore,
( ) bits/symbol
EXAMPLE 2.8 Consider a source S emits one of the three symbols S0, S1 and S3 with
their probabilities , and respectively. Find the entropy of source and list all the
elements for the second extension of this source. Also, Verify H (s2) = 2 H(s).
∑ bits/symbol
={ +
( ⁄ ) ( ⁄ )
+ ( ⁄ )
}
Therefore,
bits/symbol
The second extension of basic source s2 will have (3)2=9 symbols.
Table 1: list of probabilities
Symbol S1S1 S1S2 S1S3 S2S1 S2S2 S2S3 S3S1 S3S2 S3S3
Probability 1/4 1/8 1/8 1/8 1/16 1/16 1/8 1/16 1/16
Therefore, the entropy of the second extension source is
∑ bits/symbol
={ ( ) ( ) }
( ⁄ ) ( ⁄ ) ( ⁄ )
29
H (s2) = 3 bits/symbol
We have,
H (s2) = 2 x H(s)
=2 x 1.5 bits/symbol=3 bits/symbol
Therefore,
H (s2) = 2 H(s)
EXAMPLE 2.9 Consider a zero memory source emitting two symbols ‘X’ and ‘Y’
with respective probabilities 0.6 and 0.4. Find the entropy of source and list all the
elements for the second and third extension of this source. Also, Verify H (s3) = 3 H(s).
∑ bits/symbol
={ +
Therefore,
bits/symbol
Table 2: List of probabilities of second extension
Symbol XX XY YX YY
Probability 0.36 0.24 0.24 0.16
30
Information Theory
Therefore, the entropy of the second extension source is
∑ bits/symbol
={( )+( ) ( )}
H (s2) = 1.92 bits/symbol
Table 3: List of probabilities of third extension
Symbol XXX XXY XYX XYY YXX YXY YYX YYY
Probability 0.216 0.144 0.144 0.096 0.144 0.096 0.096 0.064
Therefore, the entropy of the third extension source is
∑ bits/symbol
= ( ) +( ) ( ) ( )
H (s3) = 2.88 bits/symbol
We have,
H (s3) = 3 x H(s)
=3 x 0.96 bits/symbol=3 bits/symbol
Therefore,
H (s3) = 3 H(s)
31
EXAMPLE 3.0 An analog baseband signal, band limited to 100 Hz, is sampled at the
Nyquist rate. The samples are quantized into four message symbols that occur
independently with probabilities P1 = P4 = 0.125 and P2 = P3. Find the information rate
(bits/sec) of the message source.
Solution: Given the probabilities P1=P4=0.125 and P2=P3=0.375. The entropy of

source is given by
∑ bits/symbol
=( ) ( )
Therefore,
bits/symbol
Since the signal is sampled at Nyquist rate, the symbol rate ‘rs’ is given by
rs=2fmax=2 x 100 = 200 symbol/sec

Information rate is given by
Rs = H(S) * rs bits/second
Rs =1.81 * 200
Therefore,
Rs =362 bits/second
1.4 Entropy of a Source with Memory

In the previous sections we have taken the information source to be memory less for
our analysis. It means that the emission of the current symbol from the source is
independent of the previous emission. But the sources which we take into account,
most among them have memory. For example, neighboring samples of the video
32
Information Theory
signal are not statistically independent. So, in real time, most of the sources are
dependent. A source with a memory can be represented by a Markov model, it is a
dependent probabilistic model required for dependent sources. For a Markov model
source, the emission of message symbols at the current instant is dependent on its
previous message symbols.
1.4.1 Mark off (Markov) Stochastic Model for Information Source

The markov’s stochastic model describes about the sequence of events in
which the probability if each occurring events depends only on the previous
event. Each event will be represented by the state diagram with their
respective probabilities. A tree diagram can be constructed by the state
diagram and the probabilities of each symbol can be determined. Stages in the
tree diagram will be dependent on the number of probabilities of the messages
to be determined.
1.4.2 Entropy and information rate of Mark off sources

Assume that the entropy of state ‘i’ is denoted by Hi . To obtain this we need to
consider all outgoing probabilities of state ‘i’ is given by
Hi = ∑ bits/symbol
………………….1.28
Where m= total number of states. The entropy of the source is given by
H=∑ bits/symbol
………………….1.29
Where is the probability of the state. The average information rate of a source is
given by,
Rs =H * rs bits/second
………………….1.30
Note: If P(mi)is the probability of a sequence ‘mi’ of ‘N’ symbols from the source is
given by,
33
GN= ∑
………………….1.31
In other words
= H bits/symbol
The value of ‘GN’ decreases for every higher length of message symbols and will be at
least equal to H.
G1 G2 ……..,…
………………….1.32
EXAMPLE 3.1 For the Markov source shown in Fig 1.3, find
(i) State probabilities
(ii) State entropies
(iii) Source entropy.
Figure 1.3: Markov Source
Solution: (i) From state diagram, the state equations are
For state A, P (A) = P (A) + P (B) + P(C)

………………….1.33
For state B, P (B) = P (B) + P (A)
………………….1.34
34
Information Theory
For state C, P(C) = P(C) + P (A)

………………….1.35
P (A) = [P (A) + P (B) + P(C)] =
P (A) =
P (B) = P (B) + P ( )
P (B) =
P (B) = P (C) + ( )
P (C) =
(ii ) The entropy of each state is
H= ∑ bits/symbol
HA = PAA PAB PAC
={ }
( ⁄ ) ( ⁄ ) ( ⁄ )
HA = 1.5 bits/symbol
HB = PBA PBB PBC
={
( ⁄ ) ( ⁄ )
}
HB = 1 bits/symbol
35
HC = PCA PCB PCC
={ }
( ⁄ ) ( ⁄ )
HC = 1 bits/symbol
(iii) The entropy of the source is given by

H=∑
= ( ) ( ) ( )
H =1.25 bits/symbol
(i) Stationary distribution
(ii) State entropies and Source entropy
(iii) Entropy of adjoint source and verify H(S) ̅
Figure 1.4: Markov Source

36
Information Theory
Solution: (i) From state diagram, the state equations are
For state A, P (A) = P (A) + P (B) + P(C)

………………….1.36
For state B, P (B) = P (A) + P (B) + P(C)
………………….1.37
For state C, P(C) = P(A) + P (B) + P(C)
………………….1.38
Subtracting equation (1.38) from equation (1.37),
P (B) - P (C) = [P (B) - P(C)]
0.7 P (B) = 0.9 P (C)
P (B) = 1.28 P (C)

………………….1.39
Multiplying equation (1.36) by 2, we get
2 P (A) = P (A) + P + P
………………….1.40
Multiplying equation (1.38) by 3, we get,
3 P (C) = P (A) + P + P
………………….1.41
Subtracting equation (1.41) from equation (1.40)
2 P (A) - 3 P (C) = P (A) - P
1.4 P (A) = 2.4 P(C)
P (A) = 1.71 P(C)

………………….1.42
But we have, P (A) + P (B) + P (C) =1
Using equation (1.42) and equation (1.39),
1.71 P(C) + 1.28 P (C) + P(C) = 1
3.99 P(C) = 1
37
P (C) = 0.25
Substituting the value of P(C) in the equation (1.39),

P (B) = 1.28 x 0.25
P(B) = 0.32
Substituting the value of P(C) in the equation (1.42),
P (A) = 1.71 x 0.25
P(A) = 0.42
(ii) The entropy of each state is

HA = PAA PAB PAC
={ }
HA = 1.37 bits/symbol
HB = PBA PBB PBC
={ }
HB = 1.48 bits/symbol
HC = PCA PCB PCC
={ }
HC = 1.57 bits/symbol
38
Information Theory
(iii) The entropy of the source is given by

H=∑
=
={ }
H =1.45 bits/symbol
………………….1.43
(iv) We know that, represents monotonically decreasing function. Hence, for the
first order of Markov source the value of N becomes unity (N=1). Entropy of
adjoint source is given by
= ∑
= ̅ =∑
= P (A) P(B) P(C)
= {0.42 0.32 }
Therefore,
̅ = 1.55 bits/symbol
………………….1.44
Comparing equation (1.43) and equation (1.44) it is proved that,
H(s) ̅
(iii) G1 and G2 and also verify G1 > G2 > H(S)
39
Figure 1.5: Markov source.
Solution: (i) From the state diagram shown in Fig. 1.5, state equations can be written
as
P (1) = 0.666 P (1) +0.25 P (2)
………………….1.45
P (2) = 0.75 P (2) +0.334 P (1)
………………….1.46
From Equation (1.45)
0.334 P(1)=0.25P(2)
P(1) =0.75P(2)
………………….1.47
But we have
P(1)+P(2) = 1
Using Equation (1.47)
0.75P(2) + P(2)=1
P(2) =0.57
P(1) =(0.75) (0.57)
P(1)=0.42
40
Information Theory
(ii) The entropy of each state is given by
For state 1:
H1 = P11 P12
={ }
H1 = 0.918 bits/symbol
Similarly for state 2:

H2 = P21 P22
={ }
Therefore, the entropy of source is given by

H=∑ bits/symbol
=P(1) H1 + P(2) H2
H= {(0.42) * (0.91) + (0.57) * (0.81)}
H=0.847 bits/symbol
………………….1.48
(iii) It is required to draw the tree diagram to find G1 and G2
41
Figure 1.6: Tree diagram showing first two states for the Markov source.
From the figure 1.6., it is recorded at the end of the first symbol interval we have, ̅ =
{A, B, C}; P (A) = 0.28; P (B) = 0.42; P (C) = (0.14+0.14) = 0.28. The entropy of source at
the end of first symbol interval is given by
GN = ∑
For N=1,
G1= H( ̅)= + + }
G1= H(̅)= 1.557 bits/symbol

………………….1.49
Similarly, the symbols obtained at the end of the second symbol interval is ̅ = {AA,
AC, CC, CB, CA, BC, BB} with probabilities {P (AA) = 0.19; P (AC) = 0.09; P (CC) =
42
Information Theory
(0.035+0.047) = 0.08; P (CB) =0.107; P (CA) = 0.095; P (BC) =0.107; P (BB) = 0.32}.
Therefore, the entropy of source at the end of second symbol interval is given by
GN= ∑
For N=2,
G2 = H( ̅ ) = { + + + +
+ }
G2= H ̅ = 1.308 bits/symbol

………………….1.50
From the equations (1.48), (1.49) and (1.50), it can easily observed that
G1> G2>H
(iii) G1 and G2 and also verify G1 > G2 > H(S)
Figure 1.7: Markov source
Solution: (i) From the state diagram shown in Fig. 1.7, state equations can be written
as
43
P (1) = ⁄ P (1) + ⁄ P (2)

………………….1.51
P (2) = ⁄ P (2) + ⁄ P (1)

………………….1.52
From Equation (1.51)
( ⁄ ) P(1) = ( ⁄ ) P(2)
P(1) =( ⁄ ) P(2)
………………….1.53
But we have
P(1)+P(2) = 1
( ⁄ ) P (2)+ P(2)=1
P(2) =

P(1) =( ⁄ )*( ⁄ )
P(1) =
(iii) The entropy of each state is given by
For state 1:
H1 = P11 P12
44
Information Theory
={ ( ) ( ) }
( ⁄ ) ( ⁄ )
Similarly for state 2:
H2 = P21 P22
= {( ) ( ) }
( ⁄ ) ( ⁄ )
Therefore, the entropy of source is given by
H=∑ bits/symbol
=P(1) H1 + P(2) H2
H= ( )*(0.984) + ( )* (0.953)
H=0.969 bits/symbol
………………….1.54
(iii) It is required to draw the tree diagram to find G1 and G2
45
Figure 1.8: Tree diagram showing first two states for the Markov source.
From the figure 1.8, it is recorded that at the end of the first symbol interval we have,
̅ = {A, B, C}; P (A) = ; P (B) = ; P (C) = ( + ) = . The entropy of source at the end
of first symbol interval is given by
GN= ∑
46
Information Theory
For N=1,
G1=H( ̅)= ( ) +( ) +( ) }
( ⁄ ) ( ⁄ ) ( ⁄ )
G1= H(̅)= 1.37 bits/symbol

………………….1.55
Similarly, the symbols obtained at the end of the second symbol interval is ̅ = {AA,
AC, CC, CB, CA, BC, BB} with probabilities {P (AA) = ; P (AC) = ; P (CC) =
( + ) = 0.357; P (CB) = ; P (CA) = ; P (BC) = ; P (BB) = }.
The entropy of source at the end of second symbol interval is given by
GN= ∑
For N=2,
G2 = H( ̅ ) = ( )
( ⁄ )
+( ) + +( ) +( ) +
( ⁄ ) ( ⁄ ) ( ⁄ )
( ) +( ) }
( ⁄ ) ( ⁄ )
G2= H ̅ = 1.302 bits/symbol
………………….1.56
From the equations (1.54), (1.55) and (1.56), it can easily observed that
G1> G2>H
47
Things to be
Remember
1. Self-information content of a 6. Entropy when the symbols are

message is obtained by equally probable is given by
H(S)max = log2q bits/sec
Ik = log2 units
7. The entropy for an nth order source
extension is given by
2. Entropy or Average information
source is obtained by
H (Sn) = n * H(S)
∑ bits/symbol
8. Source Redundancy
R =1-
3. The average rate of information or
information rate is given by
9. Entropy of a sequence „mi‟ of ‘N‟
symbols from the source after the
Rs= H(S) * rs bits/sec
end of certain intervals is given by,
4. Entropy for state „i‟ obtained with

GN=H( )= ∑
the consideration of all outgoing
probabilities and is given as
Where G1 G2 ……..,… .
5. Source Efficiency
= * 100%
48
Information Theory
DESCRIPTIVE
QUESTIONS
1. Define the following terms 7. A data frame consists of 18 fields.

(i) Amount of Information. The first and the last field of each
(ii) Average Information or Entropy. frame are same. The remaining 16
(iii) Rate of information. fields can be filled by any of the 16
(iv) Redundancy of information. symbols with equal probability.
Find the average information
2. Discuss the reasons for using conveyed by the frame. Also find
logarithmic measure for measuring the average rate of information if
information. 100 frames are transmitted every
second.
3. Give different properties of
entropy of a zero memory source. 8. Consider a zero memory source
Prove the extremal property. emitting two symbols X and Y with
respective probabilities {0.8, 0.2}.
4. A binary source is emitting Find the entropies of the source
independent sequences of 0’s and and the second order extension of
1’s with probabilities P and (1-P) the source. Also verify H (S2) =2
respectively. Plot the entropy of the H(S).
source.
9. Consider a source which emits four
5. Establish a relationship between symbols P, Q, R and S with
Harleys, Nats and Bits. respective probabilities
0.3, 0.15, 0.15 and 0.4 respectively.
6. Consider a source which emits two Calculate its efficiency and
symbols A and B. Let the redundancy.
probability of emission of A be ‘P’.
Plot the function H(s) as a function 10. For the Markov source shown in
of ‘P’. Fig. 1.9, find
(i) State probabilities.
(ii) State entropies.
49
(iii) Source Entropy.
11. For the Markov source shown in

Fig. 1.10, find
(i) State probabilities.
(ii) State entropies.
(iii) Source Entropy.
(iv) G1, G2. Also show that G1> G2> H.
12. Explain Markov model for

information source. Also discuss
the terms of entropy and
information rate of Markov source.
50
Information Theory
Multiple Choice
Questions
1. Consider a discrete memoryless (c) 2.75 bits/symbol

source with alphabet S={S0, S1, S2, S3, (d) 1.5 bits/symbol [GATE 2016, 1 M]
S4,......} and their probabilities P={ ,

Solution: The entropy of source is
, , ,....}. The entropy of the
given by
source (in bits) is
∑ bits/symbol
(a) 2 bits/symbol
(b) 1 bits/symbol = + + +
( ⁄ ) ( ⁄ ) ( ⁄ )
(c) 3 bits/symbol
(d) None of the above [GATE 2016, 2 M] ( ⁄ )
}
Solution: The entropy of source is

bits/symbol
∑ bits/symbol
={ + + +
( ⁄ ) ( ⁄ ) ( ⁄ ) 3. An analog baseband signal, band-
} limited to 100 Hz, is sampled at the
( ⁄ ) ( ⁄ )
Nyquist rate. The samples are
quantized into four message
bits/symbol
symbols that occur
with P1 = P4 =0.125 and P2 = P3.The
2. A discrete memoryless source has information rate (bits/sec) of the
an alphabet {a1, a2, a3, a4} with message source is
corresponding probabilities { , (a) 320 bits/sec
(b) 350 bits/sec
, }. The minimum required
(c) 365 bits/sec
average codeword length in bits to
(d) 362bits/sec [GATE 2016, 1 M]
represent this source for error-free
reconstruction is
Solution: P1 = P4 =0.125 and
(a) 2 bits/symbol
P2 = P3=0.375. Then the entropy of
(b) 1.75 bits/symbol
source is
51
∑ bits/symbol probability changes from equal to non

equal probabilities.
( ) ( )
5. A memoryless source emits ‘n’
symbols each with a probability ‘P’.
The entropy of the source as a function
bits/symbol of ‘n’
(a) Increases
Since the signal is sampled at Nyquist (b) Decreases as log n
rate, the symbol rate ‘rs’ is given by (c) Increases as n
rs=2 fmax=2 x 100 (d) Increases as n log n [GATE 2012: 1 Mark]
rs = 200 symbol/sec
Information rate is given by Solution: Entropy of memoryless
Rs = H(S) * rs bits/second source
Rs =1.81 * 200 H(s) =∑ bits /sym
Rs =362 bits/second Wkt., P1=P2=………….Pq =
H(s) = ∑ ( ⁄ )
=
4. A source alphabet consists of N
symbols with the probability of the Hence H(s) increases as a function
first two symbols being the same. of .
A source encoder increases the
probability of the first symbol by a 6. A source generates three symbols
small amount e. After encoding, with probability 0.25, 0.25, 0.50 at a
the entropy of the source
rate of 3000 symbols per second.
(a) Increases
Assuming independent generation
(b) Remains the same
of symbols, the most efficient source
(c) Increases only if N = 2
encoder would have average bit rate
(d) Decreases [GATE 2012: 1 Mark]
of
Solution: When the symbols are (a) 6000 bits/sec
equally probable, then the entropy of (b) 4500 bits/sec
such source results maximum and (c) 3000 bits/sec
entropy decreases when the (d) 1500 bits/sec [GATE 2006: 2 Mark]
52
Information Theory
Solution: Three symbols with H(s)max =(512 x 512 x n)

probability of 0.25, 0.25 and 0.50 at the = (512 x 512 x 3)
rate of 3000 sym/sec. Entropy of
source is given by
H(s)max = 786432
H(s) = ∑ bits /sym
= {0.25 +0.25 +0.5
8. A source produces 4 symbols with
probability { , , }. For this
source, a practical coding scheme
has an average codeword length of
2 bits/symbols. The efficiency the
Rate of information is given by code is
Rs = H(S) * rs bits/second (a) 1
Rs =1.5 * 3000 (b)
(c)
Rs =4500 bits/second
(d) [GATE 1989: 2 Mark]
7. An image uses 512× 512 picture Solution: Entropy of source is given

elements. Each of the picture by
elements can take any of the 8 H(s) = ∑ bits
distinguishable intensity levels. The /sym
maximum entropy in the above
={ +
image will be ( ⁄ ) ( ⁄ )
(a) 2097152 bits
( ⁄ ) ( ⁄ )
(b) 786432 bits
(c) 648 bits H(s) = 1.75 bits/symbol
(d) 144 bits [GATE 1990: 2 Mark]
The code efficiency of source is given

Solution: For eight distinguishable by
levels
n=log2q = log28=3 = =
Maximum entropy is given by
53
(c) H = log10 M2
= 0.875 =
(d) H = 2log10 M
9. A fair is tossed repeatedly until a 11. The information rate R for given
‘Head’ appears for the first time. Let average information H= 2.0 for
‘L’ be the number of tosses to get analog signal band limited
this first ‘Head’. The entropy H(L) to B Hz is
in bits is (a) 8 B bits/sec
(b) 4 B bits/sec
(a) 3
(c) 2 B bits/sec
(b) 1.375
(d) 16 B bits/sec
(c) 2.8
12. Entropy is
(d) 3.5 [GATE 2014: 2 Marks]
(a) Average information per
Solution: If 1 toss is required to get
message
first head, then probability P1= . If 2 (b) Information in a signal
toss are required to get first head, then (c) Amplitude of signal
probability P2= x = . If 3 toss are (d) All of the above.
required to get first head, then 13. The memory less source refers to
probability P3= x x = (a) No previous information
(b) No message storage
Entropy of source is given by
(c) independent of previous
H(s) = ∑ bits message
/sym (d) None of the above
14. The information „I‟ contained in a
={ +
( ⁄ ) ( ⁄ ) ( ⁄ ) message with probability of
= (0.5+0.5+0.375) occurrence is
(a) I = k log21/P
(b) I = k log2P
(c) I = k log21/2P
(d) I = k log21/P2
10. For M equally likely messages,
average amount of information ‘H’ 15. The expected information
(a) H = log10 M contained in a message is called
(b) H = log2 M (a) Entropy
(b) Efficiency
54
Information Theory
(c) Coded signal (c) very less

(d) None of the above (d) both a and b
16. Information rate is defined as 19. Amount of Information is

(a) Information per unit time represented by
(b) Average number of bits of (a) IK
information per second (b) PK
(c) r*H (c)
(d) All of the above
(d) H
17. If receiver knows the message 20. In discrete memoryless source, the
being transmitted, the amount of current letter produced by a source
information carried is is statistically independent of
(a) 1 (a) Past output
(b) 0 (b) Future output
(c) -1 (c) Both a and b
(d) 2 (d) None of these.
18. In more uncertainty about the

message, information carried is
(a) Less
(b) More
Answers
Q. No 1 2 3 4 5 6 7 8 9 10
Options a b d d a b b b b b
Q. No 11 12 13 14 15 16 17 18 19 20
Options b a c a a d b b a c
55
References
[1] Simon Haykin, An Introduction to Analog and Digital Communications, John Wiley
and Sons: 1989.
[2] Shanmugam K San, Digital and Analog Communication System, John Wiley and Sons:
New York; 1989.
[3] Muralidhar Kulkarni and K S Shivaprakasha, Information Theory and Coding, John
Wiley and Sons, 2015.
[4] J. S. Chitode, Information Theory and Coding, Technical Publications Pune, 2012.
[5] K. Giridhar, Information Theory and Coding, Pooja Publications.
[6] Ranjan Bose, Information Theory, Coding and Cryptography, Tata McGraw Hill: New
Delhi; 2008.
56
Chapter 2
Source Coding
Syllabus: Source coding theorem, Prefix Codes, Kraft McMillan Inequality property –
KMI. Encoding of the Source Output, Shannon’s Encoding Algorithm. Shannon Fano
Encoding Algorithm, Huffman codes, Extended Huffman coding, Arithmetic Coding, Lempel
– Ziv Algorithm.
Contents
2. Source Coding Page. No
Learning Objectives
2.1 Introduction............................................................................................................ ..................57
2.2 Types of Codes....................................................................................................... ..................58
2.2.1 Block Codes...................................................................................................................58
2.2.2 Non Singular Codes.....................................................................................................58
2.2.3 Singular Codes..............................................................................................................58
2.2.4 Uniquely Decodable Codes.........................................................................................59
2.2.5 Instantaneous Codes....................................................................................................59
2.2.6 Optimal Codes..............................................................................................................60
2.3 Prefix Codes.............................................................................................................................60
2.4 Kraft Mc-Millan Inequality...................................................................................................62
2.5 Code Efficiency and Redundancy........................................................................................64
2.6 Shannon’s Encoding Theorem..............................................................................................65
2.7 Shannon-Fano Encoding Algorithm....................................................................................81
2.7.1 Shannon-Fano Ternary Coding Algorithm..............................................................89
2.8 Compact Code: Huffman Minimum Redundancy Code..................................................92
2.9 Shannon Hartley Law and its Implications......................................................................110
2.10 Arithmetic Coding................................................................................................................112
2.10.1 Arithmetic Algorithm for decoding.......................................................................117
2.11 Lempel-Ziv Coding......................................................................................................119
2.12 Lempel-Ziv-Welsh Coding..........................................................................................122
References
57
Claude Shannon, the founder of
Information Theory, invented a way to
measure ‘the amount of information’
in a message without defining the
word information.
58
2 SOURCE CODING
Learning Objectives

Understand the need for Source Coding in the field of Communication.
Understand the different types of Codes.
Design a source encoder for various information sources.
Investigates the performance of the coding algorithms in terms of their efficiency.
Understand the compression algorithms (such as lossless and lossy) in the field of image
processing and speech processing.
2.1 Introduction
In a digital communication, the term source encoding is used prior to the information
source, where the output of information source is applied to the source encoder which
in turn, represents every information symbol emitted by the source in terms of bit
sequence called “codeword”. Now for a binary encoder, the codeword represents the
symbols in the form of (0, 1), in case of ternary code the codeword takes the symbols
in the form of (0, 1, 2). Similarly for quaternary, codeword represents the symbols in
the form of (0, 1, 2, 3) and so on. It is required to represent the data with minimum
number of bits to achieve effective communication to resolve the bandwidth scarcity
in any of the communication field.
The term source encoding refers to assigning minimal-length digital sequence to
every information symbols emitted by the source. Here, the codewords can be either
of fixed or variable length. In case of fixed length coding, the length of the codewords
is same whereas, in variable length coding, the length of the codewords is different
for the different symbols emitted from the source. Thus, a better compression can be
achieved in such codes. Compression process is applied to the source information
prior to its transmission is done to reduce the volume of information to be
transmitted. Depending upon the applications compression (such as lossless and
lossy) algorithms are carried out by the source encoder.
59
2.2 Types of Codes

In this section, we will define the types of codes where most of the codes realized in
real time are variable length codes.
2.2.1 Block Codes

In block codes each symbol will be mapped onto a block of code symbols defined in
the code alphabet. Here, the block codes can be of either fixed or variable length. For
example:
Table 2.1: Set of Codewords
A B
001 1
010 10
011 111
100 0001
The codes emitted by the source ‘A’ are of fixed length (i.e., L=3) whereas, the codes emitted
by the source ‘B’ are of variable length.
2.2.2 Non Singular Codes

All the codewords in a block code are referred as a non singular codes if they are distinct. For
example:
Table 2.2: Codeword of Symbol
A
001
011
101
111
It is observed that the codes generated by the source „A‟ are distinct from each other.
Hence it is said to be non singular code.
2.2.3 Singular Codes

A block code is said to be singular if the codes are repeated. For example:
60
Source Coding
Table 2.3: Codeword of symbol
B
001
011
001
111
It is observed that the codes generated by source „B‟ are repeated (001). Hence it is
said to be singular code.
2.2.4 Uniquely Decodable Codes:

For all finite values of „N‟ if the nth extension of a block code is non singular then a
block code is said to be uniquely decodable.
Table 2.4: Set of Codes

A B
00 1
01 10
11 111
10 110
Suppose for source „A‟ the codeword „10‟ is its nth extension and similarly for source
„B‟ codeword „110‟ is nth extension. In this case, the codes from the sources „A‟ and „B‟
are non singular (distinct). So the codes are uniquely decodable.
2.2.5 Instantaneous Codes

A uniquely code is said to be instantaneous if it is possible to recognize the end of any
codeword in a binary encoded sequence without reference to the succeeding symbols.
For example
Table 2.5: Codewords for symbols
P 001
Q 011
R 010
S 111
61
It is observed that, for the codewords 001011010111 the source symbols are P, Q, R,
and S respectively, which are unique and are of equal length 3. It is noticed from the
property of instantaneous codes that no codeword is a prefix of another codeword.
For this observation, prefix codes are sometimes also called as instantaneous codes.
2.2.6 Optimal Codes

Instantaneous codes are said to be optimal codes, if the lengths of the codeword
assigned to the symbols are of minimal length.
Table 2.6: Set of Codes
P 001 A 00
Q 011 B 01
R 010 C 10
S 111 D 11
It is observed that the source symbols P, Q, R and S are having codewords of the
length equal to 3 and the source symbols A, B, C and D are having codewords of a
length equal to 2. So it is noticed that the codewords of a source symbols A, B, C and
D are optimal codes as the assigned code length is 2.
2.3 Prefix codes

This is a variable length coding algorithm. The meaning of prefix of the codeword is
that “any sequence which is initial part of the codeword”. For example: consider a two set
of sources with encoded messages.
Table 2.7: Two set of sources
S 1
B 21
I 34
G 56
T 75
The receiver keeps decoding the information after receiving a bit stream 156213475.
Once the first bit stream „1‟ is received, the receiver decodes it as „S‟, as „1‟ is valid
codeword for „S‟. Then for the next bit stream „5‟, which is not a valid codeword hence
it accepts the next bit code „56‟ is valid codeword for „G‟. Similarly after decoding all
62
Source Coding
the bit streams the receiver gets a message as “SGBIT”. This is a valid prefix code as
the received message is same as that of transmitted message. Let us consider an
illustration for non prefix code. The receiver keeps decoding the information after
receiving a bit stream 136114475. Once the first bit stream „1‟ is received, the receiver
decodes it as „S‟, as „1‟ is valid codeword for „S‟. Then for the next bit stream „3‟, which
is not a valid codeword hence it accepts the next bit code „36‟ is valid codeword for
„G‟. For the next bit stream „1‟, even though the sent message was „11‟ for the symbol
„B‟, but it will consider „1‟ as a valid codeword for symbol „S‟. Similarly after decoding
all the bit streams the receiver gets a message as “SGSSIT”. This is a non prefix code
as the received message is not same as that of transmitted message. From the above
illustrations it is observed that for an instantaneous code no codeword should be a prefix
of any other codeword. Hence, they are also called prefix codes.
EXAMPLE 2.1 Consider symbols A, B, C, and D which gives different codes as

shown in Table 2.8. Determine which of the codes are prefix code? Also draw the
decision tree for the prefix codes.
Table 2.8: Set of codes

Symbols Code 1 Code 2 Code 3
A 0 11 10
B 10 10 110
C 111 00 111
D 1110 111 01
Solution: It is observed that code 1 is not a prefix code as the codeword “111” is also
a prefix for the codeword “1110”. In case of code 2 the codeword “11” is also a prefix
for the codeword “1110”. Hence, it is not a prefix code. Similarly, in code 3 no
codeword is a prefix of other codewords. So, it is a prefix code. Decision tree for prefix
codeword is shown in fig 2.1.
63
Fig.2.1: Decision tree for code 3
2.4 Kraft- McMillan Inequality

All prefix codes must satisfy the following property
∑ ≤1
………………….2.1
Where „r‟ is number of different symbols used in the code alphabet, „li‟ is word length
of the codeword corresponding to ith symbol and „q‟ represents number of source
symbols.
Proof: Let „ni‟ represents the number of messages encoded in to codewords of length
„i‟. Then we have
For i=1, n1 ≤ r
………………….2.2
For i=2,
n2 ≤ (r-n1) r
n2 ≤ r2- n1r
………………….2.3
In a similar way, the number of message to be encoded to 3 digits which has to be
different from code words corresponding to n1 and n2 number of messages
n3 ≤ [(r2- n1r) - n2] r or n3 ≤ r3-n1r2-n2r

………………….2.4
64
Source Coding
Proceeding this way, we can arrive at
ni ≤ ri-n1r(i-1)-n2r(i-2)……-n(i-1)r
………………….2.5
ni +n1r(i-1)+n2r(i-2)……+n(i-1)r ≤ ri
………………….2.6
Multiplying throughout by r-i, we get
ni r-i+ n1r-1+n2r-2……+n(i-1)r (i-1)≤ ri
Writing the above in reverse way, we get
ni r-i + n(i-1)r (i-1)+ n(i-2)r (i-2)+ ……+n1r -1≤ 1
∑ ≤1
………………….2.7
Since the actual number of messages „ni‟ has to be integer, we can rewrite equation 2.7
as,
∑ =( )+( ) +…+( )≤1
n1terms n2terms ni terms
………………….2.8
∑ =∑ +∑ +∑ +…+∑ ≤1
Length l1 group Length l2 group Length l3 group… Length lq group
Combining all the groups ( ) we have
∑ ≤1
The unit of word length as “BINITS” which is the short form of “BINARY DIGITS”.
For binary codes, we have r=2, so that the Kraft inequality equation 2.3 becomes,
65
∑ ≤1
2.5 Code Efficiency and Redundancy

The “average length α” of the code is given by
L=∑ binits/message symbol

………………….2.9
Where Pi = P1, P2…, Pq the respective probabilities of the source symbols S1, S2,…, Sq
and li= l1, l2,…,lq is the respective word length in binits of the codewords of the
symbols S1, S2,…,Sq. The Entropy is given by,
H(S) =∑ bits/message symbol ………………….2.10
Let us assume that,

L ≥ H(S) for binary codes
………………….2.11
L ≥ H(S) for r-array codes

………………….2.12
Where Hr(S) = entropy in r-ary units/message symbol is given by,
Hr(S) =
………………….2.13
Where „r‟ is the number of different symbols used in code alphabet. The coding
efficiency denoted by “ ” is defined in equation (2.14) and equation (2.15). For binary
codes, we have
………………….2.14
66
Source Coding
For r-ary codes, we have
………………….2.15
The coding redundancy denoted by is defined as
=1-
………………….2.16
2.6 Shannon’s Encoding Algorithm
The following steps indicate the Shannon‟s Procedure for generating binary codes.
Step 1: Note down the given source alphabet S = {S1, S2, S3, …, Sq} with their
corresponding probabilities P={ P1, P2, P3,…,Pq} for a given information source.
Where P1 ≥P2 ≥P3≥ …≥Pq.
Step 2: Now, arrange the probabilities in non-increasing order.
Step 3: Define the following parameters from the probability set
1=0
2= P1= P1+ 1
3= P2+ P1= P2+ 2
4= P3 + P2+ P1= P3+ 3
………….
………….
………….
q+1= Pq + Pq-1 +…..+ P2 + P1 = Pq + q
Step 4: Determine the smallest integer value of length „li‟ using
67
Where is the probability of symbol Si and the value of „i‟ lies in between 1 to q i.e.
i=1, 2, 3…..,q.
Step 5: Expand the decimal number „ i „in binary form till „li‟ number of places after
decimal point.
Step 6: Remove the binary point to get the desired code.
EXAMPLE 2.2 Consider symbols A, B, C and D with the corresponding probabilities

as shown in Table 2.9. Find the
(i) Codewords for the symbols by applying Shannon‟s algorithm.
(ii) Source efficiency and redundancy
Table 2.9: Set of Symbols with probabilities
Symbols A B C D
Probability ⁄ ⁄ ⁄ ⁄
Solution: Consider the following steps to generate codewords using Shannon‟s

algorithm.
Step 1: Given source alphabet S = {A, B, C, D} with their corresponding probabilities
P={ ⁄ ⁄ ⁄ ⁄ } for a given information source.
Step 2: Now, arrange the probabilities in non-increasing order as below
Table 2.10: Symbols with probabilities in non- increasing order
Symbols C D A B
Probability ⁄ ⁄ ⁄ ⁄
68
Source Coding
Step 3: Compute the sequences of „ ‟
1=0
2= ( ⁄ )= 0.5
3=( ⁄ + ⁄ )= 0.9
4=( ⁄ + ⁄ + ⁄ )= 0.96
5=( ⁄ ⁄ + ⁄ + ⁄ )= 1
Step 4: Find the smallest value of length „li‟ using
For i=1, ≥ =2, therefore, l1 =1

( ⁄ )
For i=2, ≥ =2.5, therefore, l2 =2

( ⁄ )

( ⁄ )

( ⁄ )
Step5: i „in binary form up to „li‟ number of places as

given below
1=0
2= ⁄ = (0.5)10
0.5 x 2 = 0.00 with carry 1
0.00 x 2 = 0.00 with carry 0
Therefore 2= (.10)2
3=( ⁄ + ⁄ )= (0.9)10
0.9 x 2 = 0.8with carry 1
Therefore 3= (.1110)2
69
4=( ⁄ ⁄ ⁄ )= (0.96)10
0.96 x 2 =0.92 with carry 1
0.92 x 2 =0.84 with carry 1
0.84 x 2 =0.68 with carry 1
0.68 x 2 =0.36 with carry 1
0.36 x 2 =0.72 with carry 0
Step6: Remove the binary point to get the desired code.
1=0 and l1 =1 therefore, code C =0

2= (.10)2 and l2 =2 therefore, code D =10
3= (.1110)2 and l3 =4 therefore, code A =1110
4= (.11110)2 and l4 =5 therefore, code B =11110
Now, the average „L‟ is computed by using

L= ∑ binits/ symbol
L={( ⁄ ) ( ⁄ ) ( ⁄ ) ( ⁄ ) }
L=1.75 binits/symbol
Entropy is given by,

H(s) = ∑ bits/symbol
={(( ⁄ ) ( ⁄ )
) (( ⁄ )
( ⁄ )
) (( ⁄ )
( ⁄ )
) (( ⁄ )
( ⁄
)}
)
H(S)=1.45 bits/symbol
70
Source Coding
Code Efficiency is given by

= = =0.828
Therefore percentage code efficiency
=82.8%
Code Redundancy is given by
=1- = (1- 0.828) =0.172
=17.2%
EXAMPLE 2.3 Consider symbols m1, m2, m3, m4, m5, m6, m7, and m8 with the
corresponding probabilities as shown in Table 2.11. Find the
(i) Codewords for the symbols by applying Shannon‟s algorithm.
(ii) Source efficiency and redundancy.
Table 2.11: Set of Symbols with probabilities

Symbols m1 m2 m3 m4 m5 m6 m7 m8
Probability 0.25 0.15 0.15 0.05 0.02 0.08 0.12 0.18
Solution: Consider the following steps to generate codewords using Shannon‟s

algorithm.
Step 1: Given source alphabet S ={ m1, m2, m3, m4, m5, m6, m7, m8 } with their
corresponding probabilities P={0.25,0.15,0.15,0.05,0.02,0.08,0.12,0.18} for a
given information source.
Step 2: Now, arrange the probabilities in non-increasing order as below
71
Symbols m1 m8 m2 m3 m7 m6 m4 m5
Probability 0.25 0.18 0.15 0.15 0.12 0.08 0.05 0.02

1=0
2=0.25
3= (0.25+0.18)= 0.43
4=(0.43+0.15)= 0.58
5=(0.58+0.15)= 0.73
6 =(0.73+0.12)=0.85
7 =(0.85+0.08)=0.93
8 =(0.93+0.05)=0.98
9 =(0.98+0.02)=1
Step 4: Find the smallest value of length „li‟ using

Step 5: i „in binary form up to „li‟ number of places as

given below
1=0
2= (0.25)10
72
Source Coding
0.25 x 2=0.5 with carry 0

0.5 x 2 =0 with carry 1
0 x 2 = 0 with carry 0
3= (0.43)10
0.43 x 2 =0.86
0.86 x 2 =0.72 with carry 0
0.72 x 2 =0.44 with carry 0
0.44 x 2 =0.88 with carry 0
0.88 x 2 =0.76 with carry 1
4= (0.58)10
0.58 x 2 =0.16 with carry 1
0.16 x 2 =0.32 with carry 0
0.32 x 2 =0.64 with carry 0
0.64 x 2 =0.28 with carry 1
5=( 0.73)10
0.73 x 2 =0.46 with carry 1
0.46 x 2 =0.92 with carry 0
0.92 x 2 =0.84 with carry 1
0.84 x 2 =0.68 with carry 1
0.68 x 2 =0.36 with carry 1
6 =(0.85)10
0.85 x 2 = 0.7 with carry 1

0.7 x 2 = 0.4 with carry 1
0.4 x 2 = 0.8 with carry 0
0.8 x 2 = 0.6 with carry 1
0.6 x 2 = 0.2 with carry 1
73
7= (0.85+0.08)=0.93
0.93 x 2 = 0.86 with carry 1
0.86 x 2 = 0.72 with carry 1
0.72 x 2 = 0.44 with carry 1
0.44 x 2 = 0.88 with carry 0
0.88 x 2 = 0.76 with carry 1
0.76 x 2 = 0.52 with carry 1
0.52 x 2 = 0.04 with carry 1
Therefore 7= (.1110111)2
8= (0.98)10
0.98 x 2 = 0.96 with carry 1
0.96 x 2 = 0.92 with carry 1
0.92 x 2 = 0.84 with carry 1
0.84 x 2 = 0.68 with carry 1
0.68 x 2 = 0.36 with carry 1
0.36 x 2 = 0.72 with carry 0
Therefore 8= (.1111110)2
Step 6: Remove the binary point to get the desired code.
Table 2.13: Generation of Codewords

Message Length
Probability(Pi) Codeword
symbol (li)
m1 0.25 2 00
m8 0.18 3 010
m2 0.15 3 000
m3 0.15 3 100
m7 0.12 4 1011
m6 0.08 4 1101
m4 0.05 5 11101
m5 0.02 6 111111
74
Source Coding
L= ∑ binits/symbol
L={
L=
L=3.11 binits/symbol
Entropy is given by,

={( ) ( ) ( ) ( )
( ) ( ) ( ) ( )}
H(s)=2.73 bits/symbol

= = =0.8778
=87.78%
75
=1- = (1- 0.8778) =0.1222
=12.22%
EXAMPLE 2.4 Consider symbols S1, S2 and S3 with the corresponding probabilities ,
and respectively. Find
(i) Code words for the symbols by applying Shannon‟s algorithm and also
compute the
source efficiency and redundancy.
(ii) Recompute the codewords and the efficiency with the consideration of second
order extension for the given symbols.
Solution: (i) Consider the following steps to generate codewords using Shannon‟s
algorithm.
Step 1: Given source alphabet S = {S1, S2, S3} with their corresponding probabilities P=
{ , } for a given information source.
Step 2: Now arrange the probabilities in non-increasing order as given in Table 2.14
Table 2.14: Symbols with probabilities
Symbols S2 S1 S3
Probability ⁄ ⁄ ⁄

1=0
2=( ⁄ ) 0.5
3= ( ⁄ )= 0.75
4= (0.75+ ⁄ )= 1
76
Source Coding
Step4: Find the smallest value of length „li‟ using
For i=1, ≥ =2 therefore, l1 =1

( ⁄ )
For i=2, ≥ =4 therefore, l2 =2

( ⁄ )

( ⁄ )

given below
1=0
2=
(0.5)10
0.5 x 2=0 with carry 1
Therefore 2= (.10)2
3=
(0.75)10
0.75 x 2 =0.5 with carry 1
0.5 x 2=0 with carry 1
Therefore 3= (.11)2

Message Probability
Length(li) Codeword
symbol (Pi)
S2 ⁄ 1 00
S1 ⁄ 2 10
S3 ⁄ 2 11
77
Fig.2.2: Code Tree
L={( ⁄ ) ( ⁄ ) ( ⁄ ) }
L=1.5 binits/symbol
Entropy is given by
H(S) = ∑ bits/symbol
={(( ⁄ ) ) (( ⁄ ) ) (( ⁄ ) )}
( ⁄ ) ( ⁄ ) ( ⁄ )
H(S)=1.5 bits /symbol

= = =1
=100%
78
Source Coding

=1- = (1- 1) =0
=0%
(ii) Consider the following steps for second-order extension
Step 1: Given source alphabet S ={ S1 S1, S1 S2, S1 S3, S2 S1, S2 S2, S2 S3, S3 S1, S3 S2, S3S3 }
with their corresponding probabilities P={ , , , , , , , , } for a given
information source.
Step 2: Now, arrange the probabilities in non-increasing order as below.
Symbols S2 S2 S1 S2 S2 S1 S 2 S3 S3 S2 S1 S1 S 1 S3 S3 S1 S3 S3
Probability ⁄ ⁄ ⁄ ⁄ ⁄ ⁄ ⁄ ⁄ ⁄
Step3: Compute the sequences of „ ‟
1=0
2=( ⁄ ) 0.25
3= ( ⁄ )= 0.375
4= (0.375+ ⁄ )= 0.5
5= (0.5+ ⁄ )= 0.625
6= (0.625+ ⁄ )= 0.75
7= (0.75+ ⁄ ) = 0.8125
8= (0.8125+ ⁄ ) = 0.875
9= (0.875+ ⁄ ) = 0.9375
10= (0.9375+ ⁄ )=1
Step4: Find the smallest value of length „li‟ using

79

( ⁄ )

( ⁄ )

( ⁄ )

( ⁄ )

( ⁄ )

( ⁄ )

( ⁄ )

given below
1=0
2=
(0.25)10
0.5 x 2 = 0 with carry 1
3= (0.375)10
0.375 x 2 =0.75 with carry 0
4= (0.5)10
80
Source Coding

5= (0.625)10
0.625 x 2 =0.25 with carry 1
6= (0.75)10
7= (0.8125)10
0.8125 x 2 =0.625 with carry 1
0.625 x 2=0.25 with carry 1
0.25 x 2 = 0.5 with carry 0
8= (0.875)10
0.875 x 2 =0.75 with carry 1
9= (0.9375)10
0.9375 x 2 =0.875 with carry 1
81
0.875 x 2=0.75 with carry 1

0.75 x 2 = 0.5 with carry 1
Table 2.17: Generation of codewords

Message
Probability(Pi) Length(li) Codeword
symbol
S2 S2 ⁄ 2 00
S1 S2 ⁄ 3 010
S2 S1 ⁄ 3 011
S2 S3 ⁄ 3 100
S3 S2 ⁄ 3 101
S1 S1 ⁄ 4 1100
S1 S3 ⁄ 4 1101
S3 S1 ⁄ 4 1110
S3 S3 ⁄ 4 1111
L= ∑ binits/symbol
L={( ⁄ ) ( ⁄ ) ( ⁄ ) ( ⁄ ) ( ⁄ ) ( ⁄ )
( ⁄ ) ( ⁄ ) ( ⁄ ) }
L=3 binits /symbol
Entropy is given by
82
Source Coding
={(( ⁄ ) ( ⁄ )
) (( ⁄ )
( ⁄ )
) (( ⁄ )
( ⁄ )
) (( ⁄ ) )
( ⁄ )
(( ⁄ ) ) (( ⁄ ) ) (( ⁄ ) ) (( ⁄ ) )}
( ⁄ ) ( ⁄ ) ( ⁄ ) ( ⁄ )
H(s)=3 bits/symbol

= = =1
=100%
2.7 Shannon – Fano Encoding Algorithm:

An improvement is required to increase the coding–efficiency compared to Shannon‟s
algorithm. The algorithm is as follows:
Table 2.18: Steps of Shannon-Fano Encoding Algorithm
Step 1: Arrange the probabilities in the non-increasing order.

Step 2: Grouping the probabilities must be done into exactly two sets, such that the
sum of probabilities in groups should weight equal. Now, assign bit „1‟ to
all the elements of first (top) group and bit „0‟ to the elements of second
(lower) group.
Step 3: Now, repeat step 2 till no further division is possible.
Step 4: Compute the codewords.
83
EXAMPLE 2.5 Consider symbols A, B, C, D, E and F with the corresponding

probabilities 0.15, 0.25, 0.1, 0.3, 0.12 and 0.08 respectively. Find
i) Codeword for the symbols by applying Shannon- Fano algorithm.
ii) Compute Source efficiency and redundancy.
Solution: (i) Consider the following steps to generate the codewords by applying
using Shannon-Fano approach.
Symbols D B A E C F
Probability (Pi) 0.3 0.25 0.15 0.12 0.1 0.08
Step 2: Now, assign bit „1‟ to all the elements of first (upper) group and bit „0‟ to the
elements of second (lower) group.
0.3 (1) 0.3 (1)

0.25 (1)
0.25 (0)
(0)
0.15 (0) 0.15 (1)
0.12 (0) 0.12 (0) 0.12 (1)

0.1 (0) 0.1 (0) 0.1 (0) 0.1 (1)
0.08 (0) 0.08(0) 0.08(0)
0.08 (0)
Step 3: Compute the Codewords

84
Source Coding
Table 2.20: generation of codewords
Symbols D B A E C F
Probability (Pi) 0.3 0.25 0.15 0.12 0.1 0.08
Codes 11 10 01 001 0001 0000
Length (li) 2 2 2 3 4 4
Average Length (L)

=(0.3)(2)+(0.25)(2)+(0.15)(2)+(0.12)(3)+(0.1)(4)+(0.08)(4)
= (0.6+0.5+0.3+0.36+0.4+0.32)
L =2.48 binits/symbol
ii) Entropy of the source is given by
={ + + + + + }
= (0.52+0.5+0.41+0.36+0.33+0.29)
H(s)= 2.41 bits/symbol
Efficiency of the source
= = =0.9717
=97.17%
Redundancy of the source

=1- = (1-0.9717) =0.0282
85
=2.83%
Table 2.21: Parameters Summary
Symbols D B A E C F
Probability (Pi) 0.3 0.25 0.15 0.12 0.1 0.08
Codes 11 10 01 001 0001 0000
Length (li) 2 2 2 3 4 4
Average Length ( L) L=2.48 binits/symbol
Entropy H(s) H(s)=2.41 bits/symbol
Source Efficiency ( ) =97.17%
Source Redundancy ( ) =2.83%
EXAMPLE 2.6 Consider symbols X, Y and Z with the corresponding probabilities

0.08, 0.18 and 0.74 respectively. Find
i) Codeword for the symbols by applying Shannon- Fano algorithm with
source efficiency and redundancy
ii) Compute Source efficiency and redundancy of the given symbols for 2nd
extension.
Solution: (i) Consider the following steps to generate the codewords by applying
using Shannon-Fano approach.
Symbols Z Y X
Probability (Pi) 0.74 0.18 0.08
Step 2: Now, assign bit „1‟ to all the elements of first (upper) group and bit „0‟ to the
elements of second (lower) group.
86
Source Coding
0.74 (1)
(0)
0.18 (0) 0.18 (1)
0.08 (0)
0.08 (0)
Step 3: Compute the codewords
Symbols Z Y X
Codes 1 01 00
Length (li) 1 2 2
Average Length
L1= ∑ binits/ symbol
= (0.74)(1)+(0.18)(2)+(0.08)(2)
= (0.74+0.36+0.16)
L1 =1.26 binits/symbol
Entropy of the source is given by
H(s)= ∑
={ + + }
={0.32+0.44+0.29}
H(S)= 1.05 bits/symbol
87
Efficiency of the source is

= = =0.833
=83.3%
Redundancy of the source is

=1- = (1-0.833) =0.166
=16.6%
Table 2.24: Parameters Summery
Symbols Z Y X
Codes 1 01 00
Length (li) 1 2 2
Average Length (L1) L1=1.26 binits/symbol
Source Efficiency ( ) 83.3%
Source Redundancy ( ) 16.6%
(ii) For 2nd Extension the Symbols and Probabilities are given as
Table 2.25: Symbols with probabilities of 2nd Extension
Symbols XX XY XZ YX YY YZ ZX ZY ZZ
Probability (Pi) 0.0064 0.0144 0.0592 0.0144 0.0324 0.1332 0.0592 0.1332 0.5476
88
Source Coding
Symbols ZZ YZ ZY XZ ZX YY XY YX XX
Probability (Pi) 0.5476 0.1332 0.1332 0.0592 0.0592 0.0324 0.0144 0.0144 0.0064
Step 2: Now, assign bit „1‟ to all the elements of first group (upper group) and bit „0‟
to the elements of second group (lower group).
Step 3: Compute the codewords
Probability (Pi) 0.5476 0.1332 0.1332 0.0592 0.0592 0.0324 0.0144 0.0144 0.0064
Codes 1 011 010 0011 0010 0001 00001 000001 000000
Length (li) 1 3 3 4 4 4 5 6 6
Average Length (L2) is

L2= ∑ binits/symbol
89
= {(0.5476) (1) + (0.1332) (3) (2) + (0.0592) (4) (2) + (0.0324) (4) + (0.0144) (5) + (0.0144)
(6) + (0.0064) (6)}
= (0.5476+0.7992+0.4736+0.1296+0.072+0.0864+0.0384)
H(s2) = ∑ bits/symbol
={ +( ) (2)+( )(2) + +
( )(2) + }
= (0.47+0.77+0.48+0.16+0.17+0.04)
H(s2)= 2.1 bits/symbol
Efficiency of the source is

= = =0.9813
=98.13%
Redundancy of the source is

= = (1-0.9813) =0.018
=1.87%
90
Source Coding

Probability (Pi) 0.5476 0.1332 0.1332 0.0592 0.0592 0.0324 0.0144 0.0144 0.0064
Codes 1 011 010 0011 0010 0001 00001 000001 000000
Length (li) 1 3 3 4 4 4 5 6 6
Average
L2=2.14 binits/symbol
Length (L2)
Entropy H(s2) H(s2)=2.1 bits/symbol
Efficiency
( )=98.13%
( )
Redundancy
( )=1.87%
( )
2.7.1 Shannon – Fano Ternary Encoding Algorithm

The algorithm of Shannon-Fano Ternary code is described as follows in Table 2.29
Table 2.29: Steps of Shannon-Fano Ternary code

Step 2: Grouping the probabilities must be done into exactly three sets, such that the sum of
probabilities in groups should weight equal. Now, assign bit „2‟ to all the elements of first
(top) group, assign bit „1‟ to all the elements of second (middle) group and bit „0‟ to the
elements of third (lower) group.
Step 3: Now, repeat step 2 till no further division is possible.
Step 4: compute the codewords.
EXAMPLE 2.7 Consider symbols A, B, C, D, E and F with the corresponding

probabilities 0.15, 0.3, 0.15, 0.3, 0.05 and 0.05 respectively. Find the codewords for the
symbols by applying Shannon- Fano ternary algorithm. Also compute source
efficiency and redundancy.
91
Solution: Consider the following steps to generate the codewords.

Symbols B D A C E F G
Probability (Pi) 0.3 0.3 0.15 0.15 0.05 0.03 0.02
Step 2: Now, assign bit „2‟ to all the elements of first (upper) group, bit „1‟ to the
elements of second (middle) group and bit „0‟ to the elements of third (lower)
group.
0.3 (2)
0.3 (1)
0.15 (0) 0.15 (2)

0.15 (0)
0.15(1)
0.05 (0)
0.05(0) 0.05(2)
0.03 (0)
0.03(0) 0.03(1)
0.02 (0)
0.02(0) 0.02(0)
Step 3: Compute the Codewords
Table 2.31: Generation of Codewords

Probability (Pi) 0.3 0.3 0.15 0.15 0.05 0.03 0.02
Codes 2 1 02 01 002 001 000
Length (li) 1 1 2 2 3 3 3
Average Length
92
Source Coding
=(0.3)(1)+(0.3)(1)+(0.15)(2)+(0.15)(2)+(0.05)(3)+(0.03)(3)+ (0.02)(3)
= (0.3+0.3+0.3+0.3+0.15+0.09+0.06)
iii) Entropy of the source is given by
={ + + +
+ + + }
= (0.52+0.52+0.41+0.41+0.216+0.15+0.112)
Using equation (2.15), the entropy in r-ary is given by
Hr(s) =
For r=3,
Hr(s) =
Therefore
Hr(s) = 1.476 ternary units/symbol
93
The ternary coding Efficiency

= = =0.984
=98.4%

=1- = (1-0.984) =0.016
=1.6%

Probability (Pi) 0.3 0.3 0.15 0.15 0.05 0.03 0.02
Codes 2 1 02 01 002 001 000
Length (li) 1 1 2 2 3 3 3
Average Length ( L) L=1.5 binits/symbol
Entropy for r-ary Hr(s) = 1.476 ternary units/symbol
2.8 Compact Code: Huffman Minimum Redundancy Code

The algorithm of Huffman Redundancy Code is described as follows:

Step 2: Consider the equation
q=r+(r-1)
………………….2.17
Equation (2.17) can also be written as
=
94
Source Coding
Where ‘q’ is the number of source symbols and ‘r’ is the number of different symbols used in
the code alphabet. The quantity ‘ is calculated and it should be an integer. If it is not, then
‘dummy symbols’ with zero probabilities are added to ‘q’ to make the quantity an integer.
EXAMPLE 2.8 Apply Huffman Encoding Procedure to generate binary codes for the
given symbols A, B, C, D, E and F with the corresponding probabilities 0.15, 0.2, 0.25,
0.06, 0.03 and 0.31 respectively. Also compute efficiency and redundancy of the
source code.
Solution: Using Equation (2.17) it is required to compute the value of „ ‟. For binary
code r= 2 and given source symbols (q) = 6.Therefore,
= = = 4.
Since, „ ‟ is an integer, we need not to add ant dummy symbols.
Table 2.33: Huffman Code Table
Source Probability
Stage 1 Stage 2 Stage 3 Stage 4
Symbols (Pi)
F 0.31 0.31 0.31 0.44 0.56 (0)
C 0.25 0.25 0.25 0.31 (0) 0.44 (1)
B 0.2 0.2 0.24 (0) 0.25 (1)
A 0.15 0.15 (0) 0.2 (1)
D 0.06 (0) 0.09 (1)
E 0.03 (1)
95
Symbols F C B A D E
Probability (Pi) 0.31 0.25 0.2 0.15 0.06 0.03
Codes 00 01 11 100 1010 1011
Length (li) 2 2 2 3 4 4
Average Length
=(0.31)(2)+(0.25)(2)+(0.2)(2)+(0.15)(3)+(0.06)(4)+(0.03)(4)
= (0.62+0.5+0.4+0.45+0.24+0.12)
={ + + + + + }
= (0.52+0.5+0.46+0.41+0.24+0.15)
The Coding Efficiency

= = =0.978
=97.8%
96
Source Coding

=1- = (1-0.978) =0.021
=2.1%

Symbols F C B A D E F
Probability (Pi) 0.31 0.25 0.2 0.15 0.06 0.03 0.31
Codes 00 01 11 100 1010 1011 00
Length (li) 2 2 2 3 4 4 2
Average Length ( L) L=2.33 binits/ symbol
Fig. 2.3: Code tree
EXAMPLE 2.9 Determine the efficiency of binary code by applying Huffman

Encoding Procedure for the symbols P, Q and R with the corresponding probabilities
0.62, 0.28 and 0.1 respectively. Also compute the efficiency of the binary code, if the
same technique is applied to the second order extension for the above mentions
symbols.
Solution: Using Equation (2.17) it is required to compute the value of „ ‟. Hence, for
binary code r= 2 and given that source symbols (q) = 3. Therefore
97
= = = 1.
Since, „ ‟ is an integer, we need not to add any dummy symbols.
Table 2.36: Huffman code
Source Probability
Stage 1
Symbols (Pi)
P 0.62 0.62 (0)
Q 0.28 (0) 0.39 (1)
R 0.1 (1)

Symbols P Q R
Codes 0 10 11
Length (li) 1 2 2
Average Length
=(0.62)(1)+(0.28)(2)+(0.1)(2)
= (0.62+0.56+0.2)
98
Source Coding
={ + + }
= (0.42+0.51+0.33)

= = =0.913
=91.3%

=1- = (1-0.913) =0.087
=8.7%
Symbols P Q R
Codes 0 10 11
Length (li) 1 2 2
Average Length (L) L1=1.38 binits/symbol
99
Fig.2.4. Code tree
For Second Extension the following steps are considered to generate codewords.
Symbols PP PQ PR QP QQ QR RP RQ RR
Probability
0.3844 0.1736 0.062 0.1736 0.0784 0.028 0.062 0.028 0.01
(Pi)
Arrange the probabilities of second extension symbols in non- increasing order.
Source Probability
Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6 Stage 7
Symbols (Pi)
PP 0.3844 0.3844 0.3844 0.3844 0.3844 0.3844 0.3844 0.6156 (0)
PQ 0.1736 0.1736 0.1736 0.1736 0.1736 0.2684 0.3472 (0) 0.3844 (1)
QP 0.1736 0.1736 0.1736 0.1736 0.1736 0.1736 (0) 0.2684 (1)
QQ 0.0784 0.0784 0.0784 0.124 0.1444 (0) 0.1736 (1)
PR 0.062 0.062 0.066 0.0784 (0) 0.124 (1)
RP 0.062 0.062 0.062 (0) 0.066 (1)
0.028 0.038 (0) 0.062 (1)

QR
0.028 (0) 0.028 (1)
RQ
0.01 (1)
RR
100
Source Coding
Symbols PP PQ QP QQ PR RP QR RQ RR
Probability
0.3844 0.1736 0.1736 0.0784 0.062 0.062 0.028 0.028 0.01
(Pi)
Codes 1 000 001 0100 0110 0111 01011 010100 010101
Length (li) 1 3 3 4 4 4 5 6 6
Average Length
={(0.3844)(1)+(0.1736)(3)+(0.1736)(3)+(0.0784)(4)+(0.062)(4)+(0.062)(4)+(0.028)(5)
+(0.028)(6)+(0.01)(6)}
= (0.3844+0.5208+0.5208+0.3136+0.248+0.248+0.14+0.168+0.06)
ii) Entropy of the second extended source is given by
H(s2) = 2H(s)=2 x 1.26
H(s2)=2.52 bits/symbol
= = =0.976
=97.6%
101

=1- = (1-0.976) =0.024
=2.4%
Symbols PP PQ QP QQ PR RP QR RQ RR
Probability
0.3844 0.1736 0.1736 0.0784 0.062 0.062 0.028 0.028 0.01
(Pi)
Codes 1 000 0001 00000 00001 000001 0000001 0000000 00000001
Length (li) 1 3 4 5 5 6 7 7 8
Avg. Length L2=2.6 binits/symbol
Entropy H(s) H(s2)=2.52 bits/symbol
Efficiency ( ) =97.6%
Redundancy
=2 .4%
( )
EXAMPLE 2.10 For a memoryless source (S) which emits the symbols A, B, C, D, E, F
and G with the corresponding probabilities 0.2, 0.15, 0.03, 0.05, 0.09, 0.08, and 0.4
respectively.
(i) Construct a binary Huffman Code by placing the composite symbols as
low as possible.
(ii) Generate binary Huffman Code by placing the composite symbols as
high as possible.
binary code r= 2 and given that source symbols (q) = 7. Therefore
= = = 5.
102
Source Coding
Since, „ ‟ is an integer, we need not to add any dummy symbols
Symbols A B C D E F G
Probability (Pi) 0.2 0.15 0.03 0.05 0.09 0.08 0.4
(i) Arrange the probabilities of symbols in Non- increasing order.
Table 2.44: Huffman Code
Source Probability
Stage 1 Stage 2 Stage 3 Stage 4 Stage 5
Symbols (Pi)
G 0.4 0.4 0.4 0.4 0.4 0.6 (0)
A 0.2 0.2 0.2 0.24 0.36 (0) 0.4 (1)
B 0.15 0.15 0.16 0.2 (0) 0.24 (1)
E 0.09 0.09 0.15 (0) 0.16 (1)
F 0.08 0.08 (0) 0.09 (1)
D 0.05 (0) 0.08 (1)
C 0.03 (1)
Symbols G A B E F D C
Probability (Pi) 0.4 0.2 0.15 0.09 0.08 0.05 0.03
Codes 1 000 010 011 0010 00110 00111
Length (li) 1 3 3 3 4 5 5
Average Length
=(0.4)(1)+(0.2)(3)+(0.15)(3)+(0.09)(3)+(0.08)(4)+(0.05)(5)+(0.03)(5)
= (0.4+0.6+0.45+0.27+0.32+0.25+0.15)
103
={ + + + + +
+ 0.03 }
= (0.528+0.464+0.410+0.312+0.291+0.216+0.15)

= = =0.971
=97.1%
=1- = (1-0.971) =0.029
=2.9%
Probability (Pi) 0.4 0.2 0.15 0.09 0.08 0.05 0.03
Codes 1 000 010 011 0010 00110 00111
Length (li) 1 3 3 3 4 5 5
Average Length L=2.44 binits/symbol
Redundancy ( ) =2.9%
ii) Arrange the probabilities of symbols in Non- increasing order
104
Source Coding
Source Probability
Stage 1 Stage 2 Stage 3 Stage 4 Stage 5
Symbols (Pi)
G 0.4 0.4 0.4 0.4 0.4 0.6 (0)
A 0.2 0.2 0.2 0.24 0.36 (0) 0.4 (1)
B 0.15 0.15 0.16 0.2 (0) 0.24 (1)
E 0.09 0.09 0.15 (0) 0.16 (1)
F 0.08 0.08 (0) 0.09 (1)
D 0.05 (0) 0.08 (1)
C 0.03 (1)

Probability (Pi) 0.4 0.2 0.15 0.09 0.08 0.05 0.03
Codes 1 000 010 011 0011 00100 00101
Length (li) 1 3 3 3 4 5 5
Average Length
=(0.4)(1)+(0.2)(3)+(0.15)(3)+(0.09)(3)+(0.08)(4)+(0.05)(5)+(0.03)(5)
= (0.4+0.6+0.45+0.27+0.32+0.25+0.15)

={ + + + + +
+ 0.03 }
105
= (0.528+0.464+0.410+0.312+0.291+0.216+0.15)

= = =0.971
=97.1%

=1- = (1-0.971) =0.029
=2.9%

Probability (Pi) 0.4 0.2 0.15 0.09 0.08 0.05 0.03
Codes 1 000 010 011 0011 00100 00101
Length (li) 1 3 3 3 4 5 5
Average Length L=2.44 binits/symbol
Code Tree:
Fig. 2.5: Code tree
106
Source Coding
EXAMPLE 2.11 For a memoryless source (S) which emits the symbols A, B, C, D, E
and F with the corresponding probabilities 0.01, 0.08, 0.27, 0.42, 0.06 and 0.16
respectively. Design Ternary source code using Huffman coding procedure. Also
find the efficiency and redundancy of the code.
ternary code r= 3 and given that source symbols (q) = 6.
Therefore,
= = =
Since, „ ‟ is a non integer, we need to add dummy symbols to make it an integer. Hence,
„ ‟ becomes an integer only for q=5, 7, 9, 11….and next higher integer to q=6 in the
sequence is 7, so one dummy symbol would suffice.
Symbols A B C D E F G(Dummy)
Probability (Pi) 0.01 0.08 0.27 0.42 0.06 0.16 0
Arrange the probabilities of symbols in Non- increasing order.
Source Probability
Stage 1 Stage 2
Symbols (Pi)
D 0.42 0.42 0.42(0)
C 0.27 0.27 0.31(1)
F 0.16 0.16(0) 0.27(2)
B 0.08 0.08(1)
E 0.06 (0) 0.07(2)
A 0.01 (1)
G(Dummy) 0 (2)
107
Table 2.52: Generation of Codeword
Symbols D C F B E A G(Dummy)
Probability (Pi) 0.42 0.27 0.16 0.08 0.06 0.01 0
Codes 0 2 10 11 120 121 122
Length (li) 1 1 2 2 3 3 3
Average Length
L(3)= ∑ binits/ symbol
=(0.42)(1)+(0.27)(2)+(0.16)(2)+(0.08)(2)+(0.06)(3)+(0.01)(3)+(0)(3)
= (0.42+0.54+0.32+0.16+0.18+0.03+0)
L(3) =1.65 trinits/symbol

={ + + + + + + 0}
= (0.5256+0.5100+0.4230+0.2915+0.2435+0.066+0)
The entropy for Ternary code is given by
H3(s)= =
H3(s)= 1.29 ternary units/symbol

= = =0.7818
108
Source Coding
=78.18%

=1- = (1-0.7818) =0.2182
=21.82%
Symbols D C F B E A G(Dummy)
Probability (Pi) 0.42 0.27 0.16 0.08 0.06 0.01 0
Codes 0 2 10 11 120 121 122
Length (li) 1 1 2 2 3 3 3
Average Length L(3)=1.65 trinits/symbol
Entropy H3(s) H3(s)=1.29 ternary units/symbol
EXAMPLE 2.12 For a memoryless source (S) which emits the symbols P, Q, R, S, T,
U, V and W with the corresponding probabilities 0.10, 0.18, 0.15, 0.15, 0.06, 0.04, 0.12
and 0.20 respectively. Design Quaternary source code using Huffman coding
procedure. Also find the efficiency and redundancy of the code.
Quaternary code r= 4 and given that source symbols (q) = 8.
Therefore
= = =
109
Since, „ ‟ is a non integer, we need to add dummy symbols to make it an integer. Hence,
„ ‟ becomes an integer only for q=7, 10, 13, 16….and next higher integer to q=8 in the
sequence is 10, so two dummy symbols would suffice.

Symbols P Q R S T U V W X(Dummy) Y(Dummy)
Probability
0.10 0.18 0.15 0.15 0.06 0.04 0.12 0.20 0 0
(Pi)
Arrange the probabilities of symbols in Non- increasing order.
Source Probability
Stage 1 Stage 2
Symbols (Pi)
W 0.20 0.20 0.47 (0)
Q 0.18 0.18 0.20 (1)
R 0.15 0.15 0.18 (2)
S 0.15 0.15 (0) 0.15 (3)
V 0.12 0.12 (1)
P 0.10 0.10 (2)
T 0.06 (0) 0.10 (3)
U 0.04 (1)
X 0 (2)
Y 0 (3)
Table 2.56: Generation of codeword

Symbols W Q R S V P T U X Y
Probability
0.20 0.18 0.15 0.15 0.12 0.10 0.06 0.04 0 0
(Pi)
Codes 1 2 3 00 01 02 030 031 032 033
Length (li) 1 1 1 2 2 2 3 3 3 3
110
Source Coding
Average Length
L(4)= ∑ binits/symbol
={(0.20)(1)+(0.18)(1)+(0.15)(1)+(0.15)(2)+(0.12)(2)+(0.10)(2)+(0.06)(3)
+(0.04)(3)+(0)(3)+(0)(3)}
= (0.20+0.18+0.15+0.30+0.24+0.20+0.18+0.12+0+0)
L(4) =1.57 quaternary digits/symbol
={ + +( )(2)+ + +
+ +0+ 0}
= (0.4643+0.4453+0.8210+0.3670+0.0332+0.2435+0.1857+0+0)
The entropy for Quaternary code is given by
H4(s)= =
H4(s)= 1.28 quaternary units/symbol

= = =0.8152
=81.52%
111

=1- = (1-0.8152) =0.1847
=18.47%
Symbols W Q R S V P T U X Y
Probability (Pi) 0.20 0.18 0.15 0.15 0.12 0.10 0.06 0.04 0 0
Codes 1 2 3 00 01 02 030 031 032 033
Length (li) 1 1 1 2 2 2 3 3 3 3
Average Length
L(4)=1.57 quaternary digits/symbol
L(4)
Entropy H4(s) H4(s)=1.28 quaternary units/symbol
Redundancy
=18.47%
( )
2.8 Shannon – Hartley Law and its Implications (3rd Theorem):

Shannon Hartley Law states that the capacity a band limited Gaussian channel with
AWGN is given by
C= B ( ) bits /second
Where B – channel bandwidth in Hz

S – Signal power in Watts
N– Noise power in Watts
Proof: Consider, the channel capacity „C‟ is given by
C= [H( ) – H(N)]max
………………….2.18
112
Source Coding
But we have,
H(N)max = B bits/ sec
………………….2.19
Then,
H(γ)max = B bits/ sec
………………….2.20
= (S+N)
………………….2.21
Substituting equation (2.21) in equation (2.20) we get
H(γ)max = B bits/ sec

………………….2.22
Now, substitute equation (2.22) and equation (2.19) in equation (2.18) we get
C= [B –B ]
C=
C=B
………………….2.23
EXAMPLE 2.13 A Gaussian channel has a 10MHz bandwidth. If SNR is 100, calculate
the Channel Capacity and the maximum information rate.
Solution: Given B= 10 x 106 Hz and =100. Using equation (2.23), the Channel
Capacity is obtained as
113
C= 10 x 106
C=66.58 M bits/second
Maximum Information Rate is obtained by
Rmax = B e
Rmax = (100) (10 x 106) e
Rmax =1.44 x109 bits/ second
EXAMPLE 2.14 Calculate the capacity of a Gaussian channel with a bandwidth of

1MHz and SNR is 30dB.
Solution: Given B=1x106 Hz and =30dB=103. Using equation (2.23) Channel

capacity is
C= B ( ) bits / second
= 106
C= 9.967 M bits/second
2.9 Arithmetic Coding

As discussed in the previous section 2.8, Huffman coding procedure provides good
efficiency. When the source generates large number of source alphabets containing of
many symbols with small individual probabilities. But, it fails when the source has
less number of source alphabets. To overcome the limitation, Huffman extended code
114
Source Coding
is introduced in example 2.7, which shows the issues for some extent but fails to
results optimal solution where the computation of codeword for different
combination of length for every mth extension is not possible when the values of m
increases. Hence, Arithmetic coding is introduced which generates variable length
code. It is useful when dealing with binary source having small alphabets. In this
method a unique identifier/ tag is used to distinguish a sequence of symbols from
another sequence of symbols represented by binary fraction values. The tag value is
always lies in the range of 0 to 1, where 0 denotes the lower limit and 1 denotes upper
limit. Table 2.53 shows the detailed algorithm for arithmetic coding method.
Table 2.58: General Arithmetic Algorithm for encoding the message
Step 1: Divide the numeric range 0 to 1 into number of different symbol present in the
message.
Step 2: Expand the first letter to be encoded along with the range. Further, sub-divide this
range into number of symbols.
Step 3: Repeat the procedure until termination character is encoded.
Step 4: Tag is generated in the range of [0, 1].
Step 5: Compute the total tag value of the message using
Total(message)=
To illustrate the arithmetic coding operation, let us consider the transmission of

message comprising a string of characters with the probabilities to encode in the
sequence “SGBIT”.
P(T)=0.1, P(I)=0.1, P(B)=0.25, P(G)=0.25, (S)=0.3.
115
Solution:
Fig. 2.6: Arithmetic Coding Approach
Calculation: The calculation of Arithmetic Encoding method is as follows
Step1: Divide the numeric range 0 to 1 into number of probabilities of

symbols(P(T)=0.1, P(I)=0.1, P(B)=0.25, P(G)=0.25, (S)= 0.3) in the given order.
It is required to obtain the difference value (D) using
D= (upper bound limit - lower bound limit)
The Range of character is calculated by
R (character) = lower limit + (D x P (character))
To encode the Character „S‟, the value of upper limit and lower limit is 1 and 0.7
respectively. Its difference (D) is
D= 1-0.7= 0.3
For Character „I‟, the range is
R (I) = 0.7 + (0.3 X 0.1) =0.73
116
Source Coding
For Character „T‟ the range is

R (T) = 0.73+ (0.3 X 0.1) =0.76
For Character „B‟ the range is
R (B) = 0.76+ (0.3 X 0.25) =0.835
For Character „G‟ the range is
R (G) = 0.835+ (0.3 X 0.25) =0.91
For Character „S‟ the range is
R (S) = 0.91+ (0.3 X 0.3) =1
For step2, to encode the Character „G‟, the value of upper limit and lower limit is 0.91
and 0.835 respectively. Its difference (D) is
D= 0.91-0.835= 0.075
R (I) = 0.835 + (0.075 X 0.1) =0.8425
R (T) = 0.8425+ (0.075 X 0.1) =0.85
R (B) = 0.85+ (0.075 X 0.25) =0.86875
R (G) = 0.86875+ (0.075 X 0.25) =0.8875
R (S) = 0.8875+ (0.075 X 0.3) =0.91
For step 3, to encode the Character „B‟, the value of upper limit and lower limit is
0.8685 and 0.85 respectively. Its difference (D) is
D= 0.8685-0.85= 0.01875
R (I) = 0.85 + (0.01875 X 0.1) =0.851875
117

R (T) = 0.851875+ (0.01875 X 0.1) =0.85375
R (B) = 0.85375+ (0.01875 X 0.25) =0.8584375
R (G) = 0.8584375+ (0.01875 X 0.25) =0.863125
R (S) = 0.863125+ (0.01875 X 0.3) =0.86875
For step 4, to encode the Character „I‟, the value of upper limit and lower limit is
0.851875 and 0.85 respectively. Its difference (D) is
D= 0.851875-0.85= 0.001875
R (I) = 0.85 + (0.001875 X 0.1) =0.8501875
R (T) = 0.8501875+ (0.001875 X 0.1) =0.850375
R (B) = 0.850375+ (0.001875 X 0.25) =0.85084375
R (G) = 0.85084375+ (0.001875 X 0.25) =0.8513125
R (S) = 0.8513125+ (0.001875 X 0.3) =0.851875
Finally, the value of upper limit and lower limit of Character „T‟ is 0.8501875 and
0.81875 respectively. Hence, total tag value of the String “SGBIT” is
118
Source Coding
Total (SGBIT)=
Total(SGBIT)= 0.83446875
The tag value 0.833446875 lies in between 0 to 1.
2.9.1 Arithmetic Algorithm for Decoding

For decoding the message at the receiver, it is required to get the tag value generated
during encoding approach. Table 2.59 shows the general steps for decoding the
message. Example 2.15 illustrates the complete procedure for decoding the message.
Table 2.59: General Arithmetic Algorithm for decoding the message
Step 1: Divide the numeric range 0 to 1 into number of different symbol present in the
message.
Step 2: Expand the first letter to be decoded along with the range. Further, sub-divide this
range into number of symbols with respect to tag value.
Step 3: Repeat the procedure until the tag value lies in the prescribed range.
Step 4: If the sub divided value is lies nearer to the assigned tag value, then stop the
process.
Step 5: Note down the decoded message.
EXAMPLE 2.15: Decode the message “0.572” given in the coding model with
Probabilities P(C) = 0.4, P(E)=0.5 and P(1)=0.1
Solution: The decoding process using arithmetic process is as shown in fig 2.7.
119
Fig. 2.7: Decoding Process
Calculation: The message 0.572 lies in the range of 0.4 to 0.9. For step 1, it is required
to obtain the difference value (D) using
D= (upper bound limit - lower bound limit)
The Range of character is calculated by
R (character) = lower limit + (D x P (character))
D= 0.9-0.4= 0.5
For Character „C‟, the range
R (C) = 0.4 + (0.5 X 0.4) =0.6

For Character „E‟ the range is
R (E) = 0.6+ (0.5 X 0.5) =0.85
Similarly,
R (1) = 0.85+ (0.5 X 0.1) =0.9
120
Source Coding
For step2, the 0.572 lies in the Range of 0.4 to 0.6.
D= 0.6-0.4= 0.2
For Character „C‟, the range is
R (C) = 0.4+ (0.2 X 0.4) =0.48
R (E) = 0.48+ (0.2 X 0.5) =0.58
Similarly,
R (1) = 0.58+ (0.2 X 0.1) =0.6
For step 3, it is noticed that the value 0.572 lies in the range of 0.48 to 0.58.
D= 0.58-0.48= 0.10
For Character „C‟, the range is
R(C) = 0.48+ (0.1 X 0.4) =0.52
R (E) = 0.52+ (0.1 X 0.5) =0.57
Similarly,
R (1) = 0.57+ (0.1 X 0.1) =0.58
Finally 0.572 lies in between 0.57 to 0.58. Hence, it is concluded that the decoded
message will be ECE1. Since, 0.57 and 0.58 which is the range of termination
character, stop the process.
2.10 Lempel-Ziv Coding:

In the previous section 2.9, the coding is performed only for the single character
instead of a group of character (string) so Lempel-Ziv compression algorithm is
introduced to perform coding operation for a string of characters. It is one of the
lossless compression approaches. Lempel-Ziv coding is used to compress the data by
building a dictionary of previously seen string. Table 2.60 shows the general steps for
Lempel-Ziv coding. Example 2.16 illustrates the complete procedure for Lempel-Ziv
coding.
121
Table 2.60: General steps for Lempel-Ziv coding
Step 1: The source sequence is sequentially passed in to strings that have not appeared so
far.
Step 2: After every separation we look along input sequence until we come to the shortest
string that has not been marked off before.
Step 3: We code this phrase by giving the location of the prefix and the value of the last
bit.
EXAMPLE 2.16: Obtain the codewords using Lempel-Ziv coding method for the
given string
AABABBBABAABABBBABBABBA.
Solution: Consider the following steps to generate codewords using Lempel-Ziv

method.
Step 1: The source sequence is sequentially passed in to strings that have not
appeared so far.
AABABBBABAABABBBABBABBA
Step 2: After every separation we look along input sequence until we come to the
shortest string that has not been marked off before.
Table 2.61: Dictionary consist of Content and position
Content A AB ABB B ABA ABAB BB ABBA BA

Position 1 2 3 4 5 6 7 8 9
Step 3: We code this phrase by giving the location of the prefix and the value of the
last bit. Assume the values of A=0, B=1 and null ( )=000 to generate the
codewords.
122
Source Coding

Position 1 2 3 4 5 6 7 8 9
Sequence A AB ABB B ABA ABAB BB ABBA BA
Numerical
A 1B 2B B 2A 5B 4B 3A 4A
Representation
Codeword 0000 0011 0101 0001 0100 1011 1001 0110 1000
The generated codewords for the given string is
000000110101000101001011100101101000
In order to generate a codewords, we have to look after the prefix of the last bit. For
the sequence „A‟ with position 1, it is noticed that prefix of „A‟ is null ( ).Hence, the
codeword is „0000‟. For the sequence „AB‟ with position 2, the prefix of last bit „B‟ is
„A‟ but the position for „A‟ is 1 and the generated codeword is „0011‟. Similarly for the
sequence „ABB‟ with position 3, the prefix of last bit „B‟ is „AB‟ but the position for
„AB‟ is 2 hence the generated codeword is „0101‟. The process of generating
codewords for the respective sequence continues till the position of last sequence (say
position 9).
EXAMPLE 2.17: Encode the given string 10101101101010100 using Lempel-Ziv

algorithm.
Solution: Consider the following steps to generate codewords using Lempel-Ziv

method.
Step 1: The source sequence is sequentially passed in to strings that have not
appeared so far.
10101101101010100
Step 2: After every separation we look along input sequence until we come to the
shortest string that has not been marked off before.
123
Table 2.63: Dictionary consist of Content and position

Content 1 0 10 11 01 101 010 100
Position 1 2 3 4 5 6 7 8
Step 3: We code this phrase by giving the location of the prefix and the value of the
last bit. Assume the value of null ( )=000 to generate the codewords.
Table 2.64: Generation of codeowrds

Position 1 2 3 4 5 6 7 8
Sequence 1 0 10 11 01 101 010 100
Numerical
1 0 10 11 21 31 50 30
Representation
Codeword 0001 0000 0010 0011 0101 0111 1010 0110
Therefore, the generated codewords for the given string is
00010000001000110101011110100110
In order to generate a codewords, we have to look after the prefix of the last bit. For
the sequence „1‟ with position 1, it is noticed that prefix of „1‟ is null ( ).Hence, the
codeword is „0001‟. For the sequence „0‟ with position 2, the prefix of „0‟ is null ( ) and
the generated codeword is „0000‟. Similarly for the sequence „10‟ with position 3, the
prefix of last bit „0‟ is „1‟ but the position for „1‟ is 1, hence the generated codeword is
„0010‟. The process of generating codewords for the respective sequence continues till
the position of last sequence (say position 8).
2.11 Lempel-Ziv-Welsh Coding

In the previous section 2.10 discussed, Lempel-Ziv algorithm suffers as the length of
the entries in the dictionary increases, the memory size of the dictionary has to
increase to store the data. Suppose if the memory size of the dictionary is fixed (say 4
MB) such that the entries should be done till the size of memory is full (say 4 MB). In
order to store the new entries data in the dictionary, then the previous entries data
124
Source Coding
must be erased. So to overcome the mentioned limitation, a Lempel-Ziv-Welsh coding

method is introduced. Table 2.65 shows the general steps for Lempel-Ziv-Welsh
coding. Example 2.18 illustrates the complete procedure for Lempel-Ziv-Welsh
coding.
Table 2.65: General steps for Lempel-Ziv-Welsh coding
Step 1: Create an initial dictionary with index and content of characters in a sequence.
Step 2: After every separation including previous characters, we look along input sequence
until, we come to the shortest string that has not been marked off before.
Step 3: Obtain the encoded output for the separated string of characters.
EXAMPLE 2.18: Encode the given string ababbabcababba using Lempel-Ziv-Welsh

algorithm.
Solution: Consider the following steps to generate codewords using Lempel-Ziv-

Welsh method.
Step 1: Create an Initial Dictionary which consists of index and entry of sequence.
Table 2.66: Initial Dictionary

Index Entry
1 a
2 b
3 c
Step 2: The encoding output for the above string is obtained by providing the entries
in the Table 2.67.
Fig. 2.8: Separation of sequence

125
Table 2.67 : Generation of Encoded Output

Index Entry Encoded Output
1 a
2 b
3 c
4 ab 1
5 ba 2
6 abb 4
7 bab 5
8 bc 2
9 ca 3
10 aba 4
11 abba 6
12 a 1
Therefore, the encoded output sequence for the given string ababbabcababba is
124523461
In order to generate a encoded output, it is required to construct an initial dictionary

with the characters such as a, b and c in a sequence. Now let us encode the string
where the values of a=1, b=2 and c=3 are known from the initial dictionary. For index
value 4, the sequence “ab” is occurring first time with prefix „a‟. Hence, the encoded
value is 1 (index value of „a‟ is 1). For index value 5, the sequence “ba” is occurring
first time with prefix „b‟. Hence, the encoded value is 2 (index value of „b‟ is 2).
Similarly for index value 6, the sequence “abb” is occurring first time with prefix „ab‟.
Hence, the encoded value is 4 (index value of „abb‟ is 4).The procedure continues for
remaining sequences (bab, bc, ca, aba, abba). Finally, for index value 12, the sequence
“a” is occurring with no prefix. Hence, the encoded value is 1 (index value of „a‟ is 1).
EXAMPLE 2.19: Encode the given string 101011011 using Lempel-Ziv-Welsh

algorithm.
126
Source Coding
Solution: Consider the following steps to generate codewords using Lempel-Ziv-

Welsh method.
Step 1: Create an Initial Dictionary which consists of index and entry of sequence.
Table 2.68: Initial Dictionary

Index Entry
1 1
2 0
Step 2: The encoding output for the above string is obtained by providing the entries
in the Table 2.69.
Fig. 2.9: Separation of Sequence
Table 2.69: Generation of Encoded Output

Index Entry Encoded Output
1 1
2 0
3 10 1
4 01 2
5 101 3
6 11 1
7 1011 6
8 1 1
Therefore, the encoded output sequence for the given string 101011011 is
123161
127
In order to generate a encoded output, it is required to construct an initial dictionary

with the characters such as 1 and 0 in a sequence. Now let us encode the string where
the values of 1=1, 0=2 are known from the initial dictionary. For index value 3, the
sequence “10” is occurring first time with prefix „1‟. Hence, the encoded value is 1
(index value of „1‟ is 1). Similarly for index value 4, the sequence “01” is occurring first
time with prefix „0‟. Hence, the encoded value is 2 (index value of „0‟ is 2).The
procedure continues for remaining sequences (101, 11, 1011). Finally, for index value
8, the sequence “1” is occurring with prefix. Hence, the encoded value is 1 (index
value of „1‟ is 1).
128
Source Coding
THINGS TO BE
REMEMBER
1. Different types of codes are Block 6. The Huffman codes are also called
codes, Non-singular codes, Compact codes.
uniquely decodable codes, q=r+(r-1)
Instantaneous codes and optimal =
codes. The necessary condition for an 7. The capacity a band limited Gaussian
instantaneous code is “No codeword channel with AWGN is given by
should be a prefix of any other
codeword”. Thus they are also called C= B ( ) bits /second
prefix codes.
8. Arithmetic Coding is useful when
the binary source having small
2. All prefix codes must satisfy the
alphabets. Here, a unique
Kraft Inequality property
identifier/tag is used to distinguish a
∑ ≤1
sequence of symbols from another
sequence of symbols represented by
3. Average Codeword length „L‟ is
binary fraction values. The tag value
obtained by
is always lies in the range of [0 to 1].
∑ bits/symbol
9. Lempel-Ziv algorithm is a dictionary

4. The Code efficiency denoted by is
based approach where the source
defined as
statistics are not known a priori.
=
For r-ary codes 10. Lempel-Ziv-Welsh algorithm
= supports as the length of the entries
in the dictionary increases, the
5. The Coding redundancy denoted by
memory size of the dictionary has to
is defined as
increase to store the data.
=1-
129
DESCRIPTIVE
QUESTIONS
1. Derive the expression for the code 6. Consider a zero memory source S=
efficiency and code redundancy. {S1, S2, S3, S4, S5, S6} with the
probabilities P={ 0.1, 0.2, 0.1, 0.4, 0.05,
2. Explain the steps in the Shannon‟s 0.1, 0.05}. Construct a binary Huffman
Encoding procedure for generating code by placing the composite symbol
binary codes. as high as possible. Also calculate the
variance of word length.
3. Apply Shannon‟s encoding algorithm
to generate binary codes for the set of 7. Explain the important properties of
symbols S1, S2, S3, S4 and S5 with their codes to be considered while
probabilities , , , and encoding a source.
respectively.
8. A discrete memory-less source with
4. Apply Shannon‟s Fano Coding alphabets “A to H” has respective
algorithm to generate binary codes probabilities 0.22, 0.20, 0.18, 0.15, 0.10,
for the set of symbols S= {S1, S2, S3, S4, 0.08, 0.05 and 0.02. Construct binary
S5, S6, S7} with the probabilities of P= code and ternary codes for the same
using Huffman‟s encoding algorithm.
{ , , , , , , } respectively. Also
Determine code efficiency in each
find the coding efficiency and
case.
redundancy.
9. State and explain the Shannon-
5. Consider a discrete memoryless Hartley law. Obtain an expression
source S= {S1, S2, S3, S4, S5, S6} with the for the maximum capacity of a
probabilities P={ 0.4, 0.2, 0.1, 0.1, 0.1, continuous channel.
0.05, 0.05}. Construct a binary
Huffman code by placing the 10. Explain the prefix coding and
composite symbol as low as possible. decision tree with examples.
Also calculate the variance of word
length.
130
Source Coding
11. State and explain Kraft inequality

theorem.
12. Apply Arithmetic Coding procedure

to generate a tag that lies in between
0 to 1 for the probabilities P(V)=0.4,
P(T)=0.3, P(U)=0.2, P(E)=0.08,
P(C)=0.02. Also find the total tag
value for the sequence „VTUEC‟.
13. Explain briefly the Lempel-Ziv and

Lempel-Ziv Welsh coding technique.
14. Obtain the codewords using Lempel-

Ziv coding method for the given
string XXYYXXYYYXY.
15. Encode the given string 1001101001

using Lempel-Ziv-Welsh algorithm
131
MULTIPLE CHOICE
QUESTIONS
1. The method of converting a word to (a) True

stream of bits is called as (b) False
(a) Increases (c) Both (a) and (b)
(b) Source coding
(c) Bit coding 6. In channel coding theorem, channel
(d) Cipher coding capacity decides the
_________permissible rate at which error
2. Which coding technique exhibits the free transmission is possible.
usability of fixed length codes?
(a) Lempel Ziv (a) Maximum
(b) Minimum
(b) Huffman
(c) Constant
(c) Both a and b
(d) None of the above
7. If the channel is band-limited to 6

3. When X and Y are statistically
kHz & signal to noise ratio is 16, what
independent, then I (x, y) is
would be the capacity of channel?
(a) 1
(a) 15.15 kbps
(b) 0
(b) 24.74 kbps
(c) 1 or 0 (c) 30.12 kbps
(d) Cannot be determined (d) 52.18 kbps
4. Lempel-Ziv algorithm is 8. According to Shannon's second

(a) Variable to fixed length algorithm theorem, it is not feasible to transmit
(b) Fixed to variable length algorithm information over the channel with
(c) Fixed to fixed length algorithm ______error probability, although by
(d) Variable to variable length algorithm using any coding technique.
(a) small
5. Coded system is inherently capable of (b) large
better transmission efficiency than un- (c) stable
coded system. (d) unpredictable
132
Source Coding
9. In dictionary techniques for data 10. Which bitmap file format supports
compaction, which approach of building the Run Length Encoding.
dictionary is used for the prior (a) TIFF
knowledge of probability of the (b) BMP
frequently occurring patterns? (c) PCX
(a) Static Dictionary (d) All of the above.
(b) Adaptive Dictionary
(c) Both a and b
Answers
Q. No 1 2 3 4 5
Options b a b a a
Q. No 6 7 8 9 10
Options a b a a d
References
[1] H. Jafarkhani, Space Time Coding: Theory and Practice, Cambridge University
Press, 2005.
[2] B. Vucetic and J. Yuan, Space Time Coding, Wiley, 2003.
[3] W. Stallings, Cryptography and Network Security: Principles and Practice, 4th Edition,
Prentice Hall, 2006.
[4] B. Schneier, Applied Cryptography: Protocols, Algorithms and Source Code in C, John
Wiley & Sons, 2nd Edition, 1995.
[5] M. Bloch and J. Barros, Physical Layer Security, Cambridge University Press, 2011.
[6] R. Liu and W. Trappe, Securing Wireless Communications at the Physical Layer,
Springer, 2010.
[7] Shanmugam K San, Digital and Analog Communication System, John Wiley and
Sons: New York; 1989.
[8] Muralidhar Kulkarni and K. S. Shivaprakasha, Information Theory and Coding, John
133
Delhi; 2008.
and Sons: 1989.
134
Chapter 3
Information Channels
Syllabus: Communication Channels. Channel Models, Channel Matrix, Joint probability
Matrix, Binary Symmetric Channel, System Entropies, Mutual Information, Channel
Capacity, Channel Capacity of: Binary Symmetric Channel, Binary Erasure Channel,
Muroga,s Theorem, Continuous Channels.
Contents
3. Information Channel Page. No
Learning Objectives
3.1 Introduction.............................................................................................................. ..............135
3.1.1 Representation of a Channel......................................................................................135
3.2 Joint Probability......................................................................................................... .............138
3.2.1 Properties of Joint Probability Matrix.......................................................................139
3.3 Entropy Function and Equivocation...................................................................................142
3.4 Rate of Information transmission over a discrete channel..............................................144
3.4.1 Capacity of a discrete memory less channel............................................................144
3.4.2 Channel efficiency and Redundancy........................................................................144
3.5 Special Channels....................................................................................................................145
3.5.1 Symmetric Channel.....................................................................................................145
3.5.2 Binary Symmetric Channel........................................................................................152
3.5.3 Binary Erasure Channel..............................................................................................160
3.5.4 Noiseless Channel.......................................................................................................163
3.5.5 Deterministic Channel................................................................................................165
3.5.6 Cascaded Channel.......................................................................................................167
3.6 Estimation of Channel Capacity by Muroga’s Method...................................................173
3.7 Continuous Channels...........................................................................................................179
References
135
I just wondered how
things were put together
-Claude. E. Shannon
136
3 INFORMATION CHANNELS
Learning Objectives

Understand how the communication channel affects the Communication system.
Understand the basic concepts of information theory, channel and channel capacity,
channel coding and relation among them.
Understand the basic notions of information and channel capacity.
Investigates the performance of the channel performance in terms of their efficiency.
3.1 Introduction
A channel is defined as the medium through which the coded signals produced by
an information source are transmitted. In the digital communication system the
binary data is presented at the input and the transmitted data is expected at the
receiver. Hence, the channel is considered as communication channel with various
kinds of disturbances. These disturbances may be due to its variations of amplitude
and frequency. The main characteristic of a data communication system is Channel
Capacity which represents the maximum rate at which data is transmitted with a
small probability of error across the channel. Let us see how a discrete communication
channel is completely modeled in the following section.
3.1.1 Representation of a channel

Let us consider a set of input sequence A={a1, a2, a3, ………, ar} consisting of „r‟
symbols, a set of output symbols B={ b1, b2, b3,……… bs} consisting of „s‟ symbols and
a set of conditional probabilities P( ⁄ *. Where i= 1, 2, 3,…….,r and j= 1, 2, 3, ……,s
as shown in Figure 3.1.
137
Information Theory and Coding
Fig 3.1: Representation of a channel
The conditional probabilities come into the picture due to the generation of noise at
the channel. This leads to produce the content of noise and results an uncertainty at
the receiver (loss of information). So a different number of „s‟ and „r‟ symbols are
considered at receiver and transmitter respectively. The term channel matrix or noise
matrix is represented in equation 3.1 with the arrangement of r x s number of
conditional probabilities in a matrix form, row wise input symbols and column wise
output symbols.
( ⁄ ) ( ⁄ ) ( ⁄ )
P( ⁄ * or P( ⁄ ) = ( ⁄ ) ( ⁄ ) ( ⁄ )
[ ( ⁄ ) ( ⁄ ) ( ⁄ )]
………………….3.1
Let us consider following example to illustrate the effect of noise and inter symbol
conversion. Now, consider input alphabet „A‟ emitting discrete symbols at the
transmitter with a symbols {a1, a2, a3} with a encoded binary coding as 00, 01 and 10
respectively. Because of the presence of noise in the channel receiver receives four
symbols with an alphabet B and symbols {b1,b2,b3,b4} with a codewords 00, 01, 10, and
11. Figure 3.2 illustrates “Channel Diagram” or “Noise diagram”.
138
Fig. 3.2: Channel diagram or Noise diagram
From the Channel diagram, we have

P11 = P( ⁄ )
P12 = P( ⁄ )
P13 = P ( ⁄ )
P14 = P( ⁄ )
...................3.2
From the Figure 3.2 it is viewed that when a1 (00) is transmitted and the receiver is
also expecting the same. Instead of a1 (00), either of one among four output symbols b1
(00), b2 (01), b3 (10) and b4 (11) has to be received. So, we must have
P11+ P12+ P13+ P14 =1
P( ⁄ )+ P( ⁄ )+ P ( ⁄ )+ P( ⁄ ) = 1
................3.3
If we generalize the equation (3.3) for „s‟ output symbols and „r‟ input symbols with
respect to Figure 3.2, we get
P( ⁄ )+ P( ⁄ )+ P ( ⁄ )+............+ P( ⁄ ) = 1
∑ ( ⁄ *=1
139
For the „r‟ input symbols the probabilities are P(a1), P(a2),.........., P(ar)=1
∑ =1
Total probability theorem is used to find the channel matrix elements P( ⁄ ) for all i
and j, input symbol probabilities P(ai) for all „i‟ and output symbol probabilities P(bj)
for all „j‟.
P(b1)= P( ⁄ ) P(a1)+P( ⁄ ) P(a2)+............+ P( ⁄ ) P(ar)
P(b2)= P( ⁄ ) P(a1)+P( ⁄ ) P(a2)+............+ P( ⁄ ) P(ar)
P(bs)= P( ⁄ ) P(a1)+P( ⁄ ) P(a2)+............+ P( ⁄ ) P(ar)
By considering Baye‟s rule, the input conditional probabilities P( ⁄ * is produced by
the known probabilities of P(ai) and P( ⁄ * represented in equation (3.3).
( ⁄ *
P( ⁄ *=
.........................3.3
3.2 Joint Probability
If „ai‟ and „bj‟ denotes input and output symbol, then the joint probability of both the
symbols is given by
P(ai, bj)= P( ⁄ *P( ) = P( ⁄ *P( )

.........................3.4
Now, multiply all the elements of 1st row by P(a1), 2nd row by P(a2) and rth row by P(ar)
140
( ⁄ ) ( ⁄ ) ( ⁄ )
P( ⁄ * P( )= ( ⁄ ) ( ⁄ ) ( ⁄ )
[ ( ⁄ ) ( ⁄ ) ( ⁄ ) ]
.......................3.5
Substituting equation (3.4) in equation (3.5) we get,
P(ai, bj) or P(A, B)=[ ]
........................3.6
From the equation (3.6) it is observed that the elements in the matrix are having
different joint probabilities between the input and output symbols. Such type of
arrangement matrix is called Joint Probability Matrix (JPM) represented by ( )
or
3.2.1 Properties of Joint Probability Matrix

As discussed in the previous section about the definition and the representation of
joint probability matrix in matrix form. It possesses some important properties as
follows:
Property 1: It states that the probabilities of output symbols are generated by adding
the elements of JPM column wise.
∑ ( )= ( )
........................3.7
Property 2: It states that the probabilities of output symbols can be obtained by
adding the elements of JPM in row wise.
∑ ( )=
........................3.8
Property 3: It states that the sum of all the elements of JPM is equal to unity.
∑ ∑ ( )=
........................3.9
141
EXAMPLE 3.1 Consider a set of symbols A= {a1, a2, a3} at the transmitter and a set of
symbols B= {b1, b2, b3} at the reception. For the Table 3.1, find the missing probabilities
(*) and also find P( ⁄ ) and P( ⁄ ).
Table 3.1 Probabilities of symbols

b1 b2 b3
a1 * ⁄ ⁄
a2 ⁄ ⁄ *
a3 ⁄ *
bj * ⁄ ⁄
Solution: We know that the sum of probabilities of output symbols is equal to 1.
∑ =1 and s =3
P( )+ P( )+ P( )=1
P( )+ ⁄ + ⁄ =1
P( )=1-( ⁄ )-( ⁄ )
P( )= ⁄
From the first property of JPM for the first column, we have
P( ) = P(a1, b1)+ P(a2, b1)+ P(a3, b1)
⁄ = (P(a1, b1)+ ⁄ + ⁄ )
P(a1, b1)= ⁄ - ⁄ - ⁄
P(a1, b1)= ⁄
142
For the second column

P( ) = P(a1, b2)+ P(a2, b2)+ P(a3, b2)
⁄ = ( ⁄ + ⁄ +P(a3, b2))
P(a3, b2)=( ⁄ - ⁄ - ⁄ )
P(a3, b2)= ⁄
For the third column

P( ) = P(a1, b3)+ P(a2, b3)+ P(a3, b3)
⁄ =( ⁄ + P(a2, b3) +0)
P(a2, b3)= ( ⁄ - ⁄ )
P(a2, b3)= ⁄
From the equation (3.4) of joint probability we have
P(ai, bj)= P( ⁄ *P( ) = P( ⁄ *P( )
Can be rewritten as
P( ⁄ * =
........................3.10
From the second property of JPM, for the first row
P( ) = P(a1, b1)+ P(a1, b2)+ P(a1, b3)
P( ) = ( ⁄ + ⁄ + ⁄ )
P(a1)= ⁄ and P(a1, b3)= ⁄
By substituting the values of P(a1) and P(a1, b3) in equation(3.10) we get

⁄
P( ⁄ ) =
⁄
143
P( ⁄ ) = ⁄
Equation (3.4) can be rewritten as

P( ⁄ * =
.........................3.11
Now, by substituting the value of P(a1, b1)= ⁄ and P( )= ⁄ in the equation (3.11)
we get,
⁄
P( ⁄ ) = =
⁄
P( ⁄ )= ⁄
Final Matrix obtained is

Table 3.2 Probabilities of symbols matrix
b1 b2 b3
a1 ⁄ ⁄ ⁄
a2 ⁄ ⁄ ⁄
a3 ⁄ ⁄
bj ⁄ ⁄ ⁄
3.3 Entropy Function and Equivocation

1. Priori Entropy: Priori Entropy is defined as the entropy of an input symbols a1,
a2......., ar before their transmission. It is denoted as H(A) and is given in
equation (3.12).
H(A)=∑ bits/message symbol
....................3.12
2. Posteriori Entropy (Conditional Entropy): Posteriori entropy is defined as the
entropy of the input symbols a1, a2......., ar after the transmission and output
symbol „bj‟ at the receiver. It is denoted by H( ⁄ ) and is given in equation
(3.13).
144
H( ⁄ )= ∑ ⁄ bits/message symbol
⁄
....................3.13
3. Equivocation: Equivocation is stated as the average value of all the
conditional probabilities illustrated in equation (3.13) when the value of „j‟
varies from 1 to s. It denoted by H( ⁄ ) and given by
H ( ⁄ )= E[H( ⁄ )]
Above equation can also be written as
=∑ H( ⁄ )
Where ∑ =1
Substitute the equation (3.13) in the place of H( ⁄ ), we get
=∑ ∑ ⁄
⁄
=∑ ∑ ⁄
⁄
From the equation (3.4) we have P(ai, bj)= P( ⁄ * P( ), substitute in the above
equation
H( ⁄ )=∑ ∑ bits/symbol
⁄
...........................3.14
H( ⁄ )=∑ ∑ bits/symbol
⁄
H( ⁄ ) is a measure of information at the reception hence, it is also called as the

Equivocation.
145
3.4 Rate of Information Transmission over a Discrete Channel

The average rate of information transmission Rt is given by
Rt I(A, B)rs bits/second

We know that
I(A, B)=I(B, A)=[H(A)-H( ⁄ )] =[H(B)-H( ⁄ )]
Hence,
Rt [H(A)-H( ⁄ )]rs bits/second
Or ...........................3.15
Rt [H(B)-H( ⁄ )]rs bits/second
3.4.1 Capacity of a Discrete Memory less channel

The maximum rate of information transmission possible over the channel is called as
the Capacity of channel. It is given by
C= Max {Rt}
Now, substitute equation (3.15) in the place of Rt, we get
C=Max[H(A)-H( ⁄ )]rs
...........................3.16
3.4.2 Channel Efficiency and Redundancy
The efficiency of the channel is denoted as “ “and it is given by
= x 100%
...........................3.17
Now, Substitute the value of Rt and C from the equation (3.15) and equation (3.17)
⁄
= x 100%
( ⁄ )
146
We can also rewrite the above equation as
= x 100%
..............3.18
The redundancy of the channel is denoted as “ “and it is given by
= 1-
..............3.19
3.5 Special Channels

The communication system is developed with various special channels which are
contributing lot to the system. Few among them are as follows:
i) Symmetric/ Uniform Channels
ii) Binary Symmetric Channels (BSC)
iii) Binary Erasure Channels (BEC)
iv) Noiseless Channel
v) Deterministic Channel
vi) Cascaded Channel
3.5.1 Symmetric/Uniform Channel

If the matrix of a channel is having same elements in a subsequent rows as there in the
first row in a different order, such a channel is called as Symmetric/ Uniform
Channel. Now let us consider an example
⁄ ⁄ ⁄
P( ⁄ )= ⁄ ⁄ ⁄
[ ⁄ ⁄ ⁄ ]
147
Fig. 3.2: Channel diagram
From the equation (3.14) Equivocation H (B/A) is given by,
H (B/A) = ∑ ∑ ( )
⁄
We know that
P (ai, bj) =P(ai) P( ⁄ )
H (B/A) = ∑ ∑ ⁄
⁄
H (B/A) = ∑ ∑ ⁄
⁄
Let us assume
h=∑ ⁄
⁄
We know that sum of rows is equal to 1 ( ∑ = 1). So the above equation will
become as
H (B/A) = [1] [h]

We can also rewrite as
H (B/A) =∑ ⁄
⁄
Hence, the mutual information I (A, B) is given by
I (A, B) = H (B) - H (B/A)
I (A, B) = H (B) - h
148
The Channel Capacity can be defines as

C=Max [I (A, B)]
Now, substitute the value of I (A, B)

C=Max [H (B)-h]
Therefore, the Channel Capacity „C‟ can be written as

C=Max [H (B)]-h
..................3.20
We know that the entropy of output symbol is H(B) and becomes maximum only
when every symbol at the reception will have the equi-probable values. As there are
total „S‟ output symbols and
H (B)max=
.................3.21
By substituting equation (3.21) in equation (3.20) we get,
C= – h bits/sec
.................3.22
Example 3.2: Find the Channel Capacity for the matrix given below.
P( ⁄ *=[ ]
Solution: The matrix given is belongs to a symmetric or uniform channel so, the H
(B/A) can be calculated by
H (B/A) = [h] =∑
=0.7 +0.2 +0.1

=0.36+0.46 +0.33
H (B/A) = 1.15 bits/symbol
149
The capacity of a channel is given by

C= –h
= – 1.15
C= bits/sec
Example 3.3: Calculate the capacity of the channel shown in Fig 3.3.
Solution: From the channel diagram we can write the channel matrix as below
⁄ ⁄
P( ⁄ )= ⁄ ⁄
[ ⁄ ⁄ ]
Now, to find H ( ⁄ ) we have,
H ( ⁄ ) =h=∑
= ( *+ ( ⁄
* +0
⁄
=0.31+0.5
H ( ⁄ ) = h= 0.81 bits/symbol
150
Capacity of a channel „C‟ is given by

C= –h
= – 0.81
C= 0.19 bits/symbol
Example 3.4: Find the Channel Capacity for the matrix given below.
P( ⁄ *=[ ]
with rs =1500 symbols/second
Solution: The considered matrix is of symmetric or uniform channel so, the H (B/A)
can be calculated by
H (B/A) = [h] =∑
=0.4 +0.5 +0.1
=0.52+0.5 +0.33
H (B/A) = 1.35 bits/symbol
Capacity of a channel „C‟ is given by

C= ( – h)rs
=( – 1.35) x 1500
C= 352 bits/symbol
151
Example 3.5: The following matrix of channel shows the characteristics

⁄ ⁄ ⁄
P( ⁄ ) = ⁄ ⁄ ⁄
[ ⁄ ⁄ ⁄ ]
Determine H(X), H(Y), H(X, Y) and capacity of a channel if rs =1000 symbols.
Solution: Consider that the input probabilities P(X1) = P(X2) =P(X3)= . By using
equation (3.7) of JPM calculate the columns of the matrix
P (Y1) = P ( ⁄ ) P(X1)
Then the P(X, Y) becomes
⁄ ⁄ ⁄
P = ⁄ ⁄ ⁄
[ ⁄ ⁄ ⁄ ]
From the above matrix of P(X, Y) it is observed that, the probabilities of all columns
are equal according to JPM.
P (Y1) = P (Y2) = P (Y3) = ( + + )=
The entropy of the input Probabilities is given by
H(X) = ∑
H(X) = + +
= + +
⁄ ⁄ ⁄
=0.52+0.52+0.52
H(X) =1.56 bits/symbol
152
H(Y) = ∑ ( ) ( )
H(Y) = + +
= + +
⁄ ⁄ ⁄
=0.52+0.52+0.52
H(Y) =1.56 bits/symbol
The joint Entropy H(X, Y) is given by
H(X, Y) = ∑ ∑ ( )
( )
H(X, Y) = ( )(3)+( )(6)
=0.78+2.32
H(X, Y) = 3.1 bits/symbol
Mutual Information is written as

I(X, Y) =H(X) + H(Y) - H(X, Y)
=1.56 +1.56 - 3.1
I(X, Y) =0.02
The channel capacity is given by

C=Max [I(X, Y)]
= Max [0.02]
C=0.02 bits/symbol
153
For rs =1000 symbols/second, the channel capacity in bits/second is given by
C=Max [I(X, Y)] rs

C= (0.02)(1000)
C=20 bits/second
3.5.2 Binary Symmetric Channel (BSC)

The Fig 3.4 shows the Binary Symmetric Channel, which is the most familiar and
frequently used channel in the communication system.
Fig. 3.4: BSC Channel diagram
Let us assume P(X1)=W and P(X2) = ̅ , where ̅ =1-W, P = Probability of error. It is

noted that Probability of receiver of „1‟ when „0‟ is transmitted and Probability of
receiver of „0‟ when „1‟ is transmitted. From the Fig 3.4 of BSC channel diagram the
matrix of a channel can be written as
( ⁄ ) ( ⁄ )
P( ⁄ ) = [ ]
( ⁄ ) ( ⁄ )
̅
P( ⁄ ) = * +
̅
....................3.23
154
Equation (3.23) shows the matrix of BSC channel which is symmetric in nature and
the channel will become the symmetric channel. The Equivocation of the
Symmetric/Uniform channel can be given by
H( ⁄ )=h= ∑
H( ⁄ )=h= ̅ ̅
+P
...................3.24
The Entropy of the symbols at the reception can be written as
H(Y) = ∑ ( )
( )
H(Y) = +
....................3.25
The total probability theorem can be used to find the probability of the output
symbols and at the receiver as below:
=P( ⁄ ) + P( ⁄ )
From the Fig 3.4 of BSC symmetric channel diagram we can observe that
P( ⁄ )= ̅ and =W
Similarly
P( ⁄ )=P and =̅
By substituting the above equations in P (Y1) we get,
P (Y1) = ̅ W+P ̅
To find the P(Y2),
=P( ⁄ ) + P( ⁄ )
155
From the Fig 3.4 of BSC symmetric channel diagram we can observe that
P( ⁄ )=P and =W
Similarly
P( ⁄ )= ̅ and =̅
By substituting the above equations in P (Y2) we get,
P (Y2) = W+ ̅ ̅
Now, by Substituting the values of P(Y1) and P(Y2) in equation (3.25) we get,
H(Y) = ̅ W+P ̅ ̅̅̅ ̅̅̅̅

+ W+ ̅ ̅ ̅̅
.................3.26
The mutual information of BSC can be given by
I(X, Y) =I(Y, X) = H(Y)-H ( ⁄ )
Now, by substitute the equation (3.24) and equation (3.26) in the above equation we
get,
I(X, Y) = ̅ W+P ̅ ̅̅̅ +
̅̅̅̅
W+ ̅ ̅ - [̅ +P
̅̅
]
̅
BSC is also a Symmetric/ Uniform channel, since the channel capacity is defined as
C= – h bits/ sec
C= –h
=1- h
C= 1- [ ̅ ̅ +P ] bits/sec
.................3.27
Efficiency of the channel is given by
=
156
Example 3.5: The noise matrix of a binary symmetric channel is given below, with the
transmitted symbols having probabilities of and respectively.
⁄ ⁄
P( ⁄ )=[ ]
⁄ ⁄
(i) Calculate H(X), H(Y), H(X, Y), H( ⁄ ), H( ⁄ ) and I(X, Y)

(ii) Determine the Capacity of Channel „C‟.
(iii) Find the Efficiency and redundancy of the channel.
Solution: From the equation (3.23) the channel matrix for BSC is given by
̅ ⁄ ⁄
P( ⁄ ) = * + =[ ]
̅ ⁄ ⁄
From the matrix we have P= ⁄ , ̅ = ⁄ , W= ⁄ and ̅ = ⁄
(i) The Entropy H(X) can be written as
H(X) = ∑
= ⁄ ⁄
+ ⁄ ⁄
H(X) = 1 bits/symbol
There are two output symbols P(Y1) and P(Y2) are given by
=P( ⁄ ) + P( ⁄ )
= ( ⁄ )( ⁄ ) +( ⁄ )( ⁄ )
= ⁄
157
Similarly,
=P( ⁄ ) + P( ⁄ )
=( ⁄ )( ⁄ )+( ⁄ )( ⁄ )
= ⁄
Hence, the entropy of output symbols is calculated as
H(Y) = ∑ ( )
( )
= ⁄ ⁄
+ ⁄ ⁄
H(Y) = 1 bits/symbol
From equation (3.24) the Equivocation of the Symmetric/ Uniform channel can be
given by
H( ⁄ )=h= ∑
H( ⁄ )=h= ̅ ̅
+P
= ⁄ ⁄
+ ⁄ ⁄
=0.46+0.25
H( ⁄ )=0.71 bits/symbol
The Joint Entropy H(X, Y) is written as
H(X, Y) = H(X) + H( ⁄ )
= 1+0.71
158
H(X, Y) =1.71 bits/symbol
The Joint Entropy H(Y, X) is written as
H(Y, X) =H(X, Y) = H(Y) + H( ⁄ )
H( ⁄ ) = H(X, Y) - H(Y)
=1.71 – 1
H( ⁄ ) =0.71 bits/symbol
The mutual information of BSC can be given by
I(X, Y) =I(Y, X) = H(Y) - H ( ⁄ )

=1- 0.71
I(X, Y) =0.29 bits/symbol
(ii) The capacity of a channel „C‟ id given by
C= –h
C= –H( ⁄ )
=1 - 0.71
C=0.29 bits/symbol
Example 3.6: Two independent Symbols “A” and “B” are produced by a message
source with the respective probabilities 0.7 and 0.3. Determine the source efficiency
and redundancy. If the symbols are received in an average with 8 in 100 symbols in
error, also find the transmission rate of the system.
159
Solution: Two symbols A and B with probabilities of 0.7 and 0.3 at the transmitter „X‟.
The entropy of the input symbols is
H(X) = ∑
H(X) = +
= +
=0.36+0.52
H(X) =0.88 bits/symbol
We know that
H(S) max = =
H(S)max =1 bits/symbol
Source Efficiency is given by

= =
=88%
Source Redundancy is given by =1-
=12%
From the given data, P= Probability of Error
P= =0.08
160
Therefore, P=0.08 and ̅ =0.92
It is noted that W=P (A)=0.7 and ̅ =P(B)=0.3. From the Fig 3.5 the channel matrix
can be written as
̅
P( ⁄ ) = * + =* +
̅
There are two output symbols P(Y1) and P(Y2) are given by
=P( ⁄ ) + P( ⁄ )
= +
=0.668
Similarly,
=P( ⁄ ) + P( ⁄ )
= +
=0.332
Hence, the entropy of output symbols is calculated as
H(Y) = ∑ ( ) ( )
= +
=0.38+0.52
161
H(Y) = 0.90 bits/symbol
From equation (3.24) the Equivocation of the Symmetric/ Uniform channel can be
given by
H( ⁄ )=h= ∑
H( ⁄ )=h= ̅ ̅
+P
= +
=0.11+0.29
H( ⁄ )=0.4 bits/symbol
Mutual Information I(X, Y) of BEC is,
I(X, Y) = H(Y) - H ( ⁄ )
=0.90 - 0.4
I(X, Y) =0.5 bits/symbol
Information rate Rt is given by

Rt = I(X, Y) rs
=0.5 x 100
Rt =50 bits/second
3.5.3 Binary Erasure Channel (BEC)

The BEC is used detect and correct the error. In the communication, if any erroneous
symbol occurs at the reception, in that case the symbol will be treated as „Y‟ and no
information will be considered from that received symbol. To get the correct symbol,
an immediate request will be made to retransmit the symbol through an automatic
162
request (ARQ) protocol .This process of requesting for the retransmission of the
symbol will continues till the correct symbol receives at the reception. Hence, it results
in 100% recovery of the symbols with the absence of redundancy or erroneous
symbol. This kind of a channel which produces an output without an error is called as
“Binary Erasure Channel”. The Fig 3.6 shows the diagram of BEC.
Fig. 3.6: Channel diagram of BEC
Channel matrix of the BEC is constructed from the channel diagram and is also like a
Symmetric Channel Matrix.
̅
P( ⁄ )=[ ]
̅
Let us consider P(X1) = W and P(X2) = ̅ . The Entropy of the Transmitter is given by
H(X) =∑
= +
Now, by substitute the value of P(X1) and P(X2) we get,
H(X) = W +̅ ̅
.................3.28
163
The Entropy of the symbols at the reception can be written as
H(Y) = ∑ ( )
( )
From the diagram of BEC the Value of P (Y1) and P (Y2) can be written as
P (Y1) = ̅ W + 0 and P (Y2) = 0 + ̅ ̅
P(Y) = P W + ̅ = P (W + ̅ ) = P
where W + ̅ = 1
The input Conditional Probability matrix can be written as
̅ ⁄ ⁄
̅
P( ⁄ )= P( ⁄ ) = [ ]
̅⁄ ̅̅
⁄̅ ̅
P( ⁄ )= * ̅ +
.................3.29
The equivocation, H ( ⁄ ) can be written as
H( ⁄ )=∑ ∑ ( )
⁄
=̅ + ̅
̅̅̅ +̅̅
H( ⁄ )=P[ +̅ ̅̅̅ ]
.................3.30
The Mutual Information of the BEC can be written as,
I(X, Y) = H(X)-H ( ⁄ )
164
= +̅ ̅̅̅ -P[ +̅ ̅̅̅ ]
= [1-P] [ +̅ ̅̅̅ ]
where 1-P =̅
Hence I (X, Y) can be written as

I(X, Y) = ̅ H(X)
.................3.31
Channel Capacity of the BEC can by written as
C=Max [I (X, Y)]
Substitute the equation (3.31) in above equation
= Max [̅ H(X)]
=̅ Max [ H(X)]
=̅ H(X)max
We know that H(X)max = = , the channel capacity can be written as
C=̅
This can also be written as

C= ̅
NOTE: In BEC channel “Y” will get rejected by the receiver. Hence, even though BEC
is a symmetric Channel we cannot write the mutual information as I(X, Y) = H(Y)-H
( ⁄ ).
3.5.4 Noiseless Channel

A matrix of a channel which has one and only one non-zero element in every column
is represents the “Noiseless Channel”. There are two properties of Noiseless Channel
165
Property 1: It states that there must be only one element in each column of the matrix
and the other elements should possess zero value.
Property 2: It states that the sum of any row in the matrix must be equal to unity.
Channel matrix of the noiseless channel can be written as
⁄ ⁄
P( ⁄ )=[ ⁄ ⁄ ⁄ ]
By using the logical explanation the input conditional probability matrix P ( ⁄ ) can
be written as
P( ⁄ )=[ ]
Equivocation H ( ⁄ ) is given by
H( ⁄ )=∑ ∑ ( )
⁄
We know from the equation (3.11) that,
P( ⁄ * =
Can also be written as
( )= P( ⁄ *
Now, by substitute the value of ( ) in the above equation we get
H( ⁄ )=∑ ∑ ( ⁄ *
⁄
=∑ ∑ [ ( ⁄ * ]
⁄
=1 +1 +1 +1 +1 +1
166
Therefore, Equivocation of Noiseless channel is
H( ⁄ )=0
The mutual information I (A, B) is written as
I (A, B) = H (A) - H ( ⁄ )
By Substitute the value of H ( ⁄ ) we get,

I (A, B) = H (A)
Channel capacity of Noiseless Channel can be given by,
C=Max [I (A, B)]
=Max [H (A)]
C= H (A)max =
[Since no. of inputs symbols =3].
Therefore, the channel capacity C will become
C= 1.585 bits/symbol
In general, if the no. of input symbols is „r‟ then the channel capacity can be given by,
C= bits/symbol
.................3.32
3.5.5 Deterministic Channel
A matrix of a channel which is represented by one and only one non-zero element in
every row is known as the “Deterministic Channel”.
Channel matrix of the Deterministic Channel can be written as
167
P( ⁄ )=
[ ]
Equivocation H ( ⁄ ) of a channel is given by,
H( ⁄ )=∑ ∑ ( )
⁄
We know from the equation (3.10) that

P( ⁄ * =
( ) = P( ⁄ *
Now, by substitute the value of ( ) in the above equation we get
H( ⁄ )=∑ ∑ ( )
⁄
=∑ ∑ ( ⁄ *
⁄
=1 +1 +1 +1 +1 +1
Therefore, Equivocation of Noiseless channel is
H( ⁄ )=0
The mutual information I (A, B) is written as

I (A, B) = H (B) - H ( ⁄ )
By Substitute the value of H ( ⁄ ) we get, I (A, B) = H (B)
Channel capacity of Noiseless Channel can be given by,
C=Max [I (A, B)]

168
=Max [H (B)]
C= H (B)max =
Therefore, the channel capacity C will become
C= 1.585 bits/symbol
In general, if the no. of output symbols are „s‟ then the channel capacity can be given
by,
C= bits/symbol
.................3.33
3.5.6 Cascaded Channels

When a two channels are connected in series as shown below in Fig (3.7).
Fig. 3.7: Block diagram of Cascaded Channel
Consider that the information is getting transfer from node „X‟ to the node „Y‟ through
a channel 1. There will be loss of information at the receiver node, due to the presence
of the noise in the channel 1. The mutual information at the reception of channel 1 can
be written as
I(X, Y) = H(Y)-H ( ⁄ )
.................3.34
If the same information passed to the node „Z‟ through the Channel 2, then there will
be again a more loss information. The mutual information at the reception of channel
2 can be written as
I(X, Z) = H (Z)-H ( ⁄ )
.................3.35
So, from the equation (3.34) and the equation (3.35) we can observe that
169
I(X, Z) < I(X, Y)
To determine the channel capacity of the Cascaded Channel, consider the Fig3.8
shown below
Fig. 3.8: channel diagram for BSC‟s in cascade
Let us consider,
P( ⁄ )= ̅ 2+ 2
= ̅ - 2P ̅
= 1- 2P ̅
=P( ⁄ )
Let us assume that 2P ̅ = q then
P(Z1, X1) = P(Z2, X2) = 1- q = ̅
Let us consider,
P ( ⁄ ) = P̅ +P̅ =2P ̅
=q
=P( ⁄ )
P ( ⁄ ) = P ( ⁄ )= q
170
Now, overall Channel Matrix is given by
̅
P( ⁄ )=[
̅]
For a given channel matrix, the channel diagram can be written as
Fig. 3.9: Overall channel diagram
For individual BSCs the channel capacity is written as
CXY =CYZ = 1- [ ̅ ̅ +P ]
The channel capacity of the cascade channel is
C= 1- [ ̅ ̅ +P ]
Example 3.7: Two channels which are noisy are cascaded whose channel matrix is
given as
⁄ ⁄
⁄ ⁄ ⁄
P( ⁄ )=[ ] P( ⁄ )= ⁄ ⁄
⁄ ⁄ ⁄
[ ⁄ ⁄ ]
With P(X1) =P(X2) = ⁄ , find the mutual Information I(X, Z).
171
Solution: From the matrix the channel diagram for the cascaded channel is drawn
below.
Fig. 3.10: Cascaded Channel diagram
From the Fig (3.10) of cascade channel we can write,
P( )=( ) ( ) +( ) ( )
=( ) ( )+( ) ( )=( )+( )=
( )=( ) ( ) +( ) ( )
=( ) ( )+( ) ( )=( )+( )=
( )=( ) ( ) +( ) ( )
=( ) ( )+( ) ( )=( )+( )=
( )=( ) ( ) +( ) ( )
=( ) ( )+( ) ( )=( )+( )=
( )=( ) ( ) +( ) ( )
=( ) ( )+( ) ( )=( )+( )=
172
( )=( ) ( ) +( ) ( )
=( ) ( )+( ) ( )= +( )=
From the above calculation, the equivalent channel diagram is as shown in Fig (3.11).
Fig. 3.11: Equivalent Channel diagram
With reference to the above Fig (3.11) the channel matrix can be written as
⁄ ⁄ ⁄
P( ⁄ )=[ ]
⁄ ⁄ ⁄
Consider that the input probabilities P(X1) = P(X2) = . By using equation (3.7) of JPM
calculate the columns of the matrix
P (Zk) = P ( ⁄ ) P (Xi)
Then the P(Z, X) becomes
⁄ ⁄ ⁄
P(Z, X) =[ ]
⁄ ⁄ ⁄
Entropy H ( ⁄ ) is written as,
H( ⁄ )=∑ ∑
⁄
173
={ ⁄ + ⁄ + ⁄
( ⁄ ) ( ⁄ ) ( ⁄ )
+ ⁄ + ⁄ + ⁄ }
( ⁄ ) ( ⁄ ) ( ⁄ )
=(0.21+0.44+0.26+0.02+0.38+0.19)
H ( ⁄ ) = 1.5 bits/symbol
Now, the entropy H (Z) can be calculated as
H (Z) = ∑
= + =
= + =
= + =
H (Z) ={ + + }
( ⁄ ) ( ⁄ ) ( ⁄ )
=(1.56+0.43+0.71)
H (Z)=2.7 bits/symbol
Mutual Information is given by
I(X, Z) = H(Z)- H ( ⁄ )
=(2.7 -1.5)
I(X, Z) = 1.2 bits/symbol
174
3.6 Estimation of Channel Capacity by Muroga’s Method

Let us consider a binary channel as shown in Fig (3.12) below.
Fig. 3.12: Binary Channel
Channel matrix of binary channel with reference to the Fig (3.12)
P( ⁄ )=[ ]
Dr. Muroga‟s method helps us to find the channel capacity of the binary channel with
the following steps.
Q1 and Q2 are the two quantities found using matrix equation
[ ][ ]=[ ]
By Expanding we get,
+ =
+ =
Hence, the channel capacity „C‟ can be calculated by
C= [ ] bits/symbol
Similarly, for 3x3 matrix the channel capacity can be found by,
C= bits/symbol
175
In general, the channel capacity is written as
C= [ ] bits/symbol
Example 3.8: Calculate the capacity of the channel using Muroga‟s method for the
Fig 3.13 shown below, and P(X1)=0.5, P(X2)=0.3, and P(X3)=0.2 .
Fig 3.13: Channel diagram
Solution: Matrix of the channel is constructed as
⁄ ⁄
P( ⁄ )= ⁄ ⁄
[ ⁄ ⁄ ]
The channel is a non- symmetric channel. Hence, I(X, Y) can be calculated by
I(X, Y)= H(Y)- H( ⁄ )

The channel matrix can be rewrite as,
⁄ ⁄
P(X, Y) = ⁄ ⁄
[ ⁄ ⁄ ]
176
H( ⁄ )=∑ ∑ ( )
⁄
= ⁄ + ⁄ ⁄
+ ⁄ ⁄ ⁄
+ ⁄ ⁄
⁄ ⁄
⁄ ⁄
}
=(0.15+0.25+0.09+0.15+0.1+0.1)
H ( ⁄ ) =0.84 bits/symbol
The probability of output symbol is given by JPM is,
P(Y1)= ⁄ P(X1)+ ⁄ P(X2) + ⁄ P(X3)
= ⁄ (0.5) +0+ ⁄ (0.2) =19/40

Similarly,
P(Y2)= ⁄ P(X1)+ ⁄ P(X2)+ ⁄ P(X2)
= ⁄ (0.5) + ⁄ (0.3)+ 0 = ⁄
P(Y3)= ⁄ P(X1)+ ⁄ P(X2) + ⁄ P(X3)
=0+ ⁄ (0.3)+ ⁄ (0.2) = ⁄
So, the entropy of output symbol is given by,
H (Y) = ∑ ( )
= + +
=(0.51+0.53+0.48)
H (Y) =1.52 bits/symbol
177
Therefore the mutual information I(X, Y) is
I(X, Y)=H(Y)-H( ⁄ )
=1.52-0.84
I(X, Y)=0.68 bits/symbol
The capacity of channel can be given by Muroga‟s as,
⁄ ⁄
⁄ ⁄ [ ]=[ ]
( ⁄ ) ( ⁄ ) ( ⁄ ) ( ⁄ )
[ ⁄ ⁄ ]
⁄ ⁄ =-0.311-0.5=-0.811
⁄ ⁄ =-0.311-0.5=-0.811
⁄ ⁄ =-1
By solving the above equation we get

Q1=-0.8488, Q2=-0.6976, Q3=-1.1512
The capacity of channel is

C=
C=
C=
C= C=0.68
bits/symbol
Example 3.9: A binary channel is of non-symmetric as shown in Fig (3.14) below. Find
H(X), H(Y), H(X/Y) and H(Y/X) with P(X1)=1/2 and P(X2)=1/2, α=0.65, β=0.8. Also
find the capacity of channel if the channel becomes symmetric channel α= β=0.75.
178
Solution: The channel matrix can be written as,
P(Y/X)=[ ]=* +
P(X, Y) matrix is constructed using JPM
P(X, Y)= * +
Input symbol entropy is given by
H (X) = ∑
= )
=0.5+0.5
H(X)=1 bits/symbol
To find the output symbol H(Y), we need to have P(Y1) and P(Y2)
P(Y1)=0.325+0.175=0.5
P(Y2)=0.1+0.4=0.5
H (Y) = ∑ ( )
179
= +
=(0.5+0.5)
H(Y)=1 bits/symbol
H( ⁄ )=∑ ∑ ( )
⁄
=( + + + )
=0.20+0.26+0.23+0.12
H ( ⁄ ) =0.81
We know that
H(Y)+H(X/Y)=H(X)+H(Y/X)
H(X/Y)= H(X)+H(Y/X)-H(Y)
=(1+0.81-1)
H(X/Y)=0.81 bits/symbol
When the channel becomes symmetric the capacity of the channel is
C= 1- * ̅ ̅
+
=1-* +
=1-* +
C=1-0.40+0.53=1-0.93
C=0.07 bits/symbol
180
3.7 Continuous Channels

Signals like video and telemetry are in analog. Modulation techniques such as
amplitude modulation, frequency modulation and phase modulation are in
continuous or analog form. Channel noise is considered as white Gaussian noise and
required to find the rate of transmission of information when analog signals are
contaminated with continuous noise.
181
THINGS TO BE
REMEMBER
1. The term Channel Capacity 6. Priori Entropy H (A) is given by

represents the maximum rate at
which data is transmitted with a H (A)=∑ bpms
small probability of error across the
channel.
7. Posteriori Entropy H( ⁄ ) is given
2. The input conditional by

probabilities P( ⁄ * is produced
H ( ⁄ )= ∑ ⁄
by ⁄
( ⁄ *
P( ⁄ *= 8. Equivocation H( ⁄ ) is given by
3. The probabilities of output symbols H ( ⁄ )∑ ∑

⁄
are generated by adding the bps
elements of JPM column wise. 9. In BEC channel “Y” gets rejected
by the receiver even though it‟s a
∑ ( )= ( ) symmetric channel we cannot write
the mutual information as
4. The probabilities of output symbols
can be obtained by adding the
I(X, Y) = H(Y)-H ( ⁄ )
elements of JPM in row wise.
10. The Channel capacity using

∑ ( )=
Muroga’s method is given by
C= bps
5. The sum of all the elements of JPM
is equal to Unity.
∑ ∑ ( )=
182
DESCRIPTIVE
QUESTIONS
1. Define Communication channel. 4. Two independent Symbols “P” and

With an example briefly explain “Q” are produced by a message
the representation and matrix of source with the respective
channel. probabilities 0.58 and 0.42.
Determine the source efficiency
2. Consider a set of symbols A= {a1, a2, and redundancy. If the symbols are
a3} at the transmitter and a set of received in an average with 4 in
symbols B= {b1, b2, b3} at the 100 symbols in error, also find the
reception. For the Table 3.3, find transmission rate of the system.
the missing probabilities (*) and
also find P( ⁄ ) and P( ⁄ ). 5. What is Binary Erasure Channel?
Obtain an expression for channel
Table 3.3 Probabilities of symbols capacity.
b1 b2 b3
6. Noise matrix of a binary symmetric
a1 0.7 0.1 *
channel, which has the following
a2 0.3 * 0.2
a3 * 0.4 0.6 source symbol probabilities
bj 1.0 * 1.0 P(X1)=0.42, P(X2)=0.58
3. Two noisy channels are cascaded ( ⁄ )=* +

whose channel matrices are given
by Determine H(x), H(y), H(x/y),
( ⁄ )=* + H(y/x) and I(x, y).
and
7. Find the capacity of the discrete
( ⁄ )=[ ] channel for matrix given below
With P(X1)=P(X2)=0.5. Also show

that I (X, Y) > I (X, Z).
( ⁄ )=[ ]
183
8. The noise matrix of a binary

symmetric channel is given below,
with the transmitted symbols
having probabilities of and
respectively.
P( ⁄ )=* +
(i) Calculate H(X), H(Y), H(X, Y),

H( ⁄ ), H( ⁄ ) and I(X, Y)
(ii) Determine the Capacity of
Channel „C‟.
(iii) Find the Efficiency and
redundancy of the channel.
184
MULTIPLE CHOICE
QUESTIONS
1. The capacity of a Binary Symmetric c. ( 𝑝)3

Channel (BSC) with cross-over d. 𝑝3 + 𝑝2 𝑝)
probability 0.5 is
a. Zero 4 During transmission over a certain
b. Two binary communication channel, bit
c. One errors occur independently with
d. Three probability p. The probability of at
most one bit in error in a block of n
2. In a digital communication system, bits is given by
transmission of successive bits a. 𝑝𝑛
through a noisy channel is assumed b. 𝑝𝑛
to be independent events with error c. ( 𝑝)
probability p. The probability of at d. ( 𝑝) + (
1 𝑝)𝑛
most one error in the transmission
of an 8-bit sequence is 5. A communication channel with
a. (1 − 𝑝)8 + (1 − 𝑝)7 AWGN operating at a signal at a
b. (1 − 𝑝)8 + 8 (1 − 𝑝)8 signal to noise ratio SNR >>1 and
c. (1 − 𝑝)8 + 𝑝(1 − 𝑝)7 bandwidth B has capacity C1. If the
d. (1 − 𝑝)8 + 8 (1 − 𝑝)7 SNR is doubled keeping B
constant, the resulting capacity C2
3. Consider a Binary Symmetric is given by
Channel (BSC) with probability of a. 𝐶2 ≈ 2𝐶1
error being p. To transmit a bit say b. 𝐶2 ≈ 𝐶1 + 0.3
1, we transmit a sequence of three c. 𝐶2 ≈ 𝐶1 +
sequences to represent 1 if at least
d. 𝐶2 ≈ 𝐶1 + 2
two bits bit will be represent in error
is
6. In channel coding theorem, channel
a. 𝑝3 + 3𝑝2 𝑝)
capacity decides
b. 𝑝3
the………permissible rate at which
error free transmission is possible.
185
a. Maximum c. Both (a) and (b)

b. Minimum d. None of the above
c. Constant 9. The sum of all the elements of JPM is
d. All the above equal to
a. Unity
7. Which type of channel does not b. Zero
represent any correlation between c. Two
input and output symbols? d. None of the above
a. Noiseless channel
b. Lossless channel 10. If a noiseless channel band-limited
c. Deterministic channel to 5 KHz is sampled every 1msec,
d. Useless channel what will be the value of sampling
frequency?
8. The probabilities of output symbols a. 250 samples/sec
are generated by b. 500 samples/sec
a. adding elements of JPM column c. 800 samples/sec
wise. d. 1000 samples/sec
b. adding the elements of JPM row
wise.
Answers
Q. No 1 2 3 4 5
Options a d a d c
Q. No 6 7 8 9 10
Options a d c a d
References
[1] T.M. Cover and J. A. Thomas, Elements of information theory, John Wiley & Sons,
2012.
[2] A. B. Robert, Information Theory, Dover Special Priced Titles, 2007.
[3] R. M. Roth, Introduction to Coding Theory, Cambridge University Press, 2006.
New York; 1989.
186
Delhi; 2008.
and Sons: 1989.
187
Work Space
188
Chapter 4
Error Control Coding

Syllabus: Introduction, Examples of Error control coding, methods of Controlling Errors,
Types of Errors, types of Codes, Linear Block Codes: matrix description of Linear Block Codes,
Error Detection and Error Correction Capabilities of Linear Block Codes, Single Error
Correcting hamming Codes, Table lookup decoding using Standard Array.
Contents
4. Error Control Coding Page. No
Learning Objectives
4.1 Introduction...........................................................................................................................187
4.2 Examples of Error Control Coding.....................................................................................187
4.3 Methods of controlling errors..............................................................................................188
4.4 Types of errors................................................................................................................ .......189
4.5 Types of Codes.......................................................................................................................189
4.6 Linear Block Codes......................................................................................................... .......189
4.7 Matrix Description of linear block codes...........................................................................190
4.8 Encoding circuit for (n, k) linear block codes....................................................................192
4.9 Syndrome Calculation Circuit.............................................................................................193
4.10 Distance Properties of linear block codes..........................................................................206
4.11 Single error correcting Hamming Codes...........................................................................213
4.12 Table look-up decoding using the standard array...........................................................218
References
189
We know the past but cannot
control it. We control the future
but cannot know it.
Claude. E. Shannon
-Claude. E.
190
4 ERROR CONTROL CODING
Learning Objectives

Understand the need for error control codes in the Communication system.
Understand the different types of error control codes.
Silent features of linear block codes.
Design and analyze the encoding and decoding process of linear block codes.
Special class of linear block codes such as Hamming codes is discussed.
4.1 Introduction
In previous chapters, many various kinds of coding approaches are used to obtain
lower value of average length that in turn achieves higher efficiency. These coding
techniques suffer from two issues such as variable length and fixed length. So to
overcome the limitations the process of detecting and correction codes “Error Control
Coding” is introduced which provides systematic addition of redundant symbols
with affecting the designed system. In this chapter, the meaning of error control
coding and various methods to detect and correct the errors that help to provide
higher efficiency are discussed.
4.2 Example of Error Control Coding

Error Control Coding is used to improve the data quality by the proper use of
redundancy bits in the communication. The two functional blocks such as channel
encoder and decoder at the transmitter and receiver respectively that accomplish
error control coding. Because of this reason, error control coding is also considered as
Channel Encoding. Fig 4.1 shows the building blocks of digital communication
system that employs error control coding. The main blocks of the digital
communication systems are Channel encoder, Channel decoder and Communication
Channel with capacity „C‟ bits/sec.
191
Now let us consider the source which generates a message block {bk} at a rate of ‘rb’
bits/sec. The message block is fed to the channel encoder which adds (n-k) number of
redundant bits to these „k’ bit messages generated from the source to form ‘n’ bit code
words.
Fig. 4.1: Building blocks of digital communication system
These additional (n-k) bits are called check bits. Where these check bits carries no
information but helps to detect and correct errors at the channel decoder. The
message block { k} received at the receiver is differs from the transmitted block.
Hence the probability of error is denoted by Pe=P( k bk). The value of ‘Pe’ must be
less than prescribed value.
4.3 Methods of Controlling Errors

In order to control the errors in any communication system two methods are noticed
such as Forward-acting Error Correction and Error Detection method. In case of
Forward-acting Error Correction method, the errors are controlled at the receiver
through many attempts to reduce noise. But in case of Error Detection method, the
decoder examines the received sequence. It accepts only when the received sequence
matches with a valid sequence and rejects when the received sequence is matched and
a notification will be sent to the transmitter through reverse channel about the error
and requests to transmitter for retransmission of the message till the correct sequence
received by the reception. Hence the decoder detects the errors but it does not attempt
to correct the errors. But due to the use of reverse channel, it slows down the effective
rate of data transmission.
192
4.4 Types of Errors

Since Gaussian noise and impulse noise are the noise generated at the communication
channel in communication system. Due to the existence of these noises, the errors
such as Random Error and Burst Error occur that affects the communication process.
Random Errors occurs due to the presence of white Gaussian noise. Whereas Burst
Error occurs due to the presence of impulse noise generated from lightning, manmade
noise, switching transients etc., which affects more than one symbol and generates the
error.
4.5 Types of Codes

Error control codes are divided into two categories namely Block code that consists of
(n-k) number of redundant check bits that are added to ‘k’ number of information bits
to produce ‘n’ bit codewords. Whereas, in Convolution codes the check bits are
continuously interleaved with the information bits that helps to correct the errors in
other blocks also.
4.6 Linear Block Codes

Let us consider a block of ‘k’ message bits treated as an input sequence which is
encoded to form ‘n’ bit codewords with the addition of (n-k) number of check bits as
shown in Fig 4.2.
Fig. 4.2: Formation of linear block code
A code is said to be a (n-k) linear block code if it satisfies two important characteristics
such as
193
(i) If C1 and C2 be any of the codewords which belongs to the set of (n, k) block
code, then C1+C2 is also a ‘n’ bit codeword that belongs to the same set of (n, k)
block code.
(ii) A (n-k) linear block code is systematic, if the „k’ message bits appear either at
the beginning or end of the codeword.
4.7 Matrix Description of Linear Block Codes

Let us consider the codewords generated from the message block of „k’ bits can be
represented as a row-vector or k-tuple called Message vector and is given in equation
4.1
[D]= {d1, d2,......., dk}
………………….4.1
Where d1, d2,......., dk are either 0’s or 1’s. The channel encoder systematically adds (n-
k) check bits to generate (n, k) linear block code. The code vector (2k) can be
represented by
C = {C1, C2,......., Cn}
………………….4.2
In case of Systematic linear block code, the message block of „k‟ bits will appear at the
beginning of the code vector and is given by
Ci = di
………………….4.3
Where i=1, 2, 3, ….., k. The remaining (n-k) bits are check bits. Hence equation (4.2) and
equation (4.3) can be combined as
[C] = {C1, C2,......., Cn, Ck+1, Ck+2,……, Cn}

………………….4.4
Where C1, C2,......., Cn are k message bits and ck+1, ck+2,……, cn are (n-k) check bits and
are derived from k message bits using predetermined rule as given in equation (4.5).
Ck+1= P11d1+ P21d2+………..+ Pk1dk
Ck+2= P12d1+ P22d2+………..+ Pk2dk
Cn = P1, n-kd1+ P2, n-kd2+…+ Pk, n-kd2

………………….4.5
194
Where P11, P21, P12, P22 … are either 0‟s or 1‟s and performed addition using modulo-2
arithmetic. The process of modulo-2 arithmetic is given in Table 4.1.
Table 4.1: Modulo-2 Arithmetic Procedure

Modulo-2 Addition Modulo-2 Multiplication Modulo-2 Subtraction
0+0=0 0.0=0 0-0=0
0+1=1 0.1=0 0-1=1
1+0=1 1.0=0 1-0=1
1+1=0 1.1=1 1-1=0
Now it is required to combine equations (4.3), (4.4) and (4.5) and express the results in
a matrix form as
[C1, C2,......., Cn, Ck+1, Ck+2,……, Cn]= [d1, d2,......., dk]
[ ]
………………….4.6
Equation (4.6) can be rewritten as
[C] = [D] [G]
………………….4.7
Where [G] is known as “Generation matrix” of order (kxn) and is given by
[G] = [Ik P](kxn)

………………….4.8
Where Ik= unit matrix of order „k‟ and [P] Parity matrix of order kx(n-k). Hence, the
generator matrix [G] can also expressed as
[G] = [P Ik]
195
The parity check matrix with order (n-k)xn is defined in equation (4.8)
[G] = [PT In-k]

………………….4.9
Where PT indicates parity check matrix.
4.8 Encoding Circuit for (n, k) Linear Block Codes

From equations (4.1), (4.2) and (4.5) we can have
C1=d1
C2=d2
Ck=dk
Ck+1= P11d1+ P21d2+………..+ Pk1dk
Ck+2= P12d1+ P22d2+………..+ Pk2dk
Cn = P1, n-kd1+ P2, n-kd2+…+ Pk, n-kd2

………………….4.10
In order to implement the equation (4.10) in a circuit requires an encoder for (n, k)
linear block code. The realization of encoder circuit consisting of a k-bit shift register
and modulo-2 adder is shown in Fig 4.3.
Fig. 4.3: Encoder Circuit for (n, k) linear block code

196
4.9 Syndrome Calculation Circuit

Consider the code vector c= (c1, c2 …cn) which is transmitted over a noisy
communication channel. Because of the noisy channel, the received vector may be a
corrupted version of the transmitted code vector. The function of the decoder is to
determine the transmitted code vector based on the received vector. In case of linear
block codes the decoder attempts initially to compute the syndrome to check the
received code vector is a valid or invalid code. Now let us consider the received code
vector be r = (r1, r2…rn-1). Then the syndrome vector S = [s1, s2, ….sn-k] is given in
equation (4.11)
[S] = [R] [H]T
………………….4.11
Therefore
S = [s1, s2, ….sn-k] = [r1, r2, ….rn]
[ ]
………………….4.12
Syndrome bits are obtained by multiplying received vector with the matrix using
modulo-2 arithmetic.
S1= r1P11+r2P21+……+rkPk1+rk+1
S2= r1P12+r2P22+……+rkPk2+rk+2
Sn-k= r1P1,n-k+r2P2,n-k+……+rkPk,n-k+rn
………………….4.13
In order to implement the equation (4.13) in a circuit requires a Syndrome calculation
circuit for linear block code. The realization of Syndrome calculation circuit consisting
of a k-bit shift register and modulo-2 adder is shown in Fig 4.4.
197
Fig. 4.4: Generation of Syndrome Circuit using received vector
The error pattern „E‟ is obtained as the difference between R and C is given in
equation (4.14). i.e., E=R-C or E=R+C. Therefore
C=R+E
………………….4.14
Example 4.1: For a (6, 3) linear block code, the parity matrix „P‟ is given by
P=[ ]
Find the possible code vectors. Also draw the encoding and decoding circuit for the
transmitted and received vectors respectively.
198
Solution: For a (6, 3) linear block code, i.e., n=6 and k=3. Hence it generates 2k=23=8
message vectors. The code vectors are formed using equation (4.7) i.e., [C] = [D] [G].
But from equation (4.8) we have
[G] = [ Ik P]
Where „Ik‟ denotes identity matrix for the different values of „k‟. Since k=3, then the
generation matrix [G] is
[G] = [I3 P]
[G]=[ ]
Now it is possible to generate the code vectors using equation (4.7). Therefore
[C]=[d1, d2, d3] [ ]
Table 4.2 provides the detailed information of systematic (6, 3) linear block code for
the 2k message vectors.
Table 4.2: Code vector for (6, 3) linear block code
Message Vector [D] Code Vector [C] for (6,3) linear block code
C4=d1 +
d1 d2 d3 C1=d1 C2=d2 C3=d3 C5=d1 + d2 C6=d2 + d3
d3
0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 1
0 1 0 0 1 0 0 1 1
0 1 1 0 1 1 1 1 0
1 0 0 1 0 0 1 1 0
1 0 1 1 0 1 0 1 1
1 1 0 1 1 0 1 0 1
1 1 1 1 1 1 0 0 0
199
Now for the possible code vector [C], the code vector bits are [C1=d1, C2=d2, C3=d3, C4=d1
+ d3, C5=d1 + d2, C6=d2 + d3]. It is required to draw the encoding circuit for (6, 3) linear
block code. Since k=3, it is necessary to consider three shift register to shift the
message bits. Also requires (n-k)=(6-3)=3 modulo-2 adders to construct the circuit and
is shown in Fig 4.5.
Fig. 4.5: Encoding circuit for (6, 3) linear block code.
For constructing the syndrome calculation circuit for the (6, 3) linear block code, it is
necessary to consider equation (4.11)
[S]= [R] [H]T
We know the definition
[H]=[In-k PT]
………………….4.15
From the above equation (4.15), the transpose of parity check matrix [H] can be
obtained in equation (4.16). Therefore
[H]T [ ]= [ ]
………………….4.16
Where „I3‟ denotes identity matrix. Therefore
[H]T=
[ ]
200
As we know that from the above transpose of parity check matrix [H]T, the received
vector is R=[r1, r2, r3, r4, r5, r6]. Therefore
[S]= [R] {H]T
[s1, s2, s3]= [r1, r2, r3, r4, r5, r6] [H]T
[s1, s2, s3]= [r1, r2, r3, r4, r5, r6]
[ ]
Therefore
[s1, s2, s3]= [r1+r3+r4, r1+r2+r5, r2+r3+r6]
Hence the Syndrome bits are tabulated in table 4.3.
Table 4.3: Syndrome Bits
Syndrome Bits
s1 r1+r3+r4
s2 r1+r2+r5
s3 r2+r3+r6
By considering the values of syndrome bits, it is possible to construct the syndrome

calculation circuit and is shown in Fig 4.6.
Fig. 4.6: Syndrome calculation circuit for (6, 3) linear block code
201
P=[ ]
Find the possible valid code vectors. Also draw the encoding and decoding circuit for
the transmitted and received vectors respectively. A single error has occurred in each
of the received vectors. And also detect and correct the errors (i) Rp=[0110010] (ii)
Rq=[1111101].
Solution: For a (7, 4) linear block code, i.e., n=7 and k=4. Hence it generates 2k
message vectors. The code vectors are formed using equation (4.7) i.e., [C] = [D] [G].
[G] = [ Ik P]
where „Ik‟ denotes identity matrix for the different values of „k‟. Since k=4, then the
[G] = [ I4 P]
[G]=[ ]
[C]=[d1, d2, d3, d4] [ ]
the 2k=24=16 message vectors
202

Message Vector
Code Vector [C] for (7, 4) linear block code
[D]
d1 d2 d3 d4 C1=d1 C2=d2 C3=d3 C4=d4 C5=d1+d3+d4 C6=d1+d2+d4 C7=d2+d3+d4
0 0 0 0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 1 1 1 1
0 0 1 0 0 0 1 0 1 0 1
0 0 1 1 0 0 1 1 0 1 0
0 1 0 0 0 1 0 0 0 1 1
0 1 0 1 0 1 0 1 1 0 0
0 1 1 0 0 1 1 0 1 1 0
0 1 1 1 0 1 1 1 0 0 1
1 0 0 0 1 0 0 0 1 1 0
1 0 0 1 1 0 0 1 0 0 1
1 0 1 0 1 0 1 0 0 1 1
1 0 1 1 1 0 1 1 1 0 0
1 1 0 0 1 1 0 0 1 0 1
1 1 0 1 1 1 0 1 0 1 0
1 1 1 0 1 1 1 0 0 0 0
1 1 1 1 1 1 1 1 1 1 1
Now for the possible code vector [C], the code vector bits are [C1=d1, C2=d2, C3=d3, C4=d4,
C5=d1+d3+d4, C6=d1+d2+d4, C7=d2+d3+ d4]. It is required to draw the encoding circuit for (7,
4) linear block code. Since k=4, it is necessary to consider three shift register to move
the message bits. Also requires (n-k)=(7-4)=3 modulo-2 adders to construct the circuit
and is shown in Fig 4.7.
Fig. 4.7: Encoding circuit for (7, 4) linear block code

203
For constructing the syndrome calculation circuit for the (7, 4) linear block code, it is
necessary to consider equation (4.11)
[S]= [R] [H]T
We know the definition
[H]=[In-k PT]
From the above, the transpose of parity check matrix [H] can be obtained in equation
(4.17). Therefore
[H]T [ ]= [ ]
………………….4.17
Where „I3‟ denotes identity matrix. Therefore
[H]T=
[ ]
As we know that from the above [H]T matrix, the received vector is R=[r1, r2, r3, r4, r5,
r6, r7]. Therefore
[S]= [R] {H]T
[s1, s2, s3]= [r1, r2, r3, r4, r5, r6, r7] [H]T
[s1, s2, s3]= [r1, r2, r3, r4, r5, r6, r7]
[ ]
Therefore
[s1, s2, s3]= [r1+r3+r4+r5, r1+r2+r4+r6, r2+r3+r4+r7]
Hence the syndrome bits are given in Table 4.5.
204

Syndrome Bits
s1 r1+r3+r4+r5
s2 r1+r2+r4+r6
s3 r2+r3+r4+r7
By considering the values of syndrome bits, it is possible to construct the syndrome

calculation circuit and is shown in Fig 4.8.
Fig. 4.8: Syndrome calculation circuit for (7, 4) linear block code
(i) For the received vector Rp=[0110010]
The syndrome „Sp‟ is given as

Sp= Rp [H]T
Sp= [0110010]
[ ]
Therefore
Sp= [011]
205
Since the syndrome Sp= [011] is located in second row of transpose of parity check
matrix [H]T. The corresponding error vector is
Ep = [0100000]
The corrected code vector is given by

Cp= Rp + Ep
Cp= [0110010] + [0100000]
Cp= [0010010]
(ii) For the received vector Rq=[0110010]
The syndrome „Sp‟ is given as

Sp= Rp [H]T
Sp= [1111100]
[ ]
Therefore
Sp= [010]
Since the syndrome Sp= [010] is located in sixth row of transpose of parity check
matrix [H]T. The corresponding error vector is
Eq = [0000010]
The corrected code vector is given by

Cq= Rq + Eq
206
Cq= [1111100] + [0000010]
Cq= [1111110]
Example 4.3: If „C‟ is a valid code vector then prove that CHT=0 where HT is the
transpose of the parity check matrix „H‟.
Solution: Consider the ith row of the [G] is defined in equation (4.18)
Gi = [0 0…1…0 Pi1 Pi2 … Pij… Pi, n-k]

………………….4.18
Similarly the jth row of the [H] is given in equation (4.19)
Hj= [P1j P2j … Pij… Pkj 0 0…1…0]
………………….4.19
Now using equations (4.18) and (4.19) we get
Gi HjT = [0 0…1…0 Pi1 Pi2 … Pij… Pi, n-k]
[ ]
Therefore Gi HjT= Pij + Pij= Pij(0+0)
This implies Gi HjT= 0

………………….4.20
Pre multiplying by [D] both sides on equation (4.20), we get
207
[D][Gi] HjT= [D] [0]
But we know that [C] = [D] [G]. Therefore
[C] [H]T=0
Example 4.4: For the given generation matrix of (5, 1) code [G] = [1 1 1 1 1]. Find the
Parity check matrix [H] and evaluate the syndrome for the received vector.
Solution: For (5, 1) code, there are only one message bit and the remaining three bits
are check bits. We know that,
[G] = [P Ik] = [P I1]
[G] = [1 1 1 1 1]
Therefore
[P] = [1 0 0 1]
The parity check matrix with respect to generator matrix is
[H] = [In-k PT] = [I4 PT]
Therefore
[H] = [ ]
Since k=1, there are only two valid code vectors and the message vectors [D] can
either be [0] or [1]. Therefore
[C]= [0] [G] = [0] [1 1 1 1 1] = [0 0 0 0 0]
[C]= [1] [G] = [1] [1 1 1 1 1] = [1 0 0 1 1]

208
Suppose if the transmitted vector be [0 0 0 0 0]. Then there are five possible single
error patterns are generated such as [1 0 0 0 0], [0 1 0 0 0], [0 0 1 0 0], [0 0 0 1 0] and [0 0
0 0 1]. The syndrome for the single error pattern is generated using equation (4.11).
[S] = [R] [H]T
For the received vector [0 1 0 0 0]
S1=[1 0 0 0 0]
[ ]
S1=[1 0 0 0 ]
S2=[0 1 0 0 0]
[ ]
S2=[0 1 0 0 ]
S3=[0 0 1 0 0]
[ ]
S3=[0 0 1 0 ]
209
S4=[0 0 1 0 0]
[ ]
S4=[0 0 0 1 ]
S5=[0 0 0 0 1]
[ ]
S5=[1 1 1 1 ]
The syndrome bits are tabulated in table 4.6.

Syndrome Bits
S1 [1 0 0 0 ]
S2 [0 1 0 0 ]
S3 [0 0 1 0 ]
S4 [0 0 0 1 ]
S5 [1 1 1 1 ]
4.10 Distance Properties of a Linear Block Code

In this section, the concept of „weight‟ and „distance‟ between code-vectors are
discussed and was first introduced by R. W. Hamming.
210
4.10.1 Hamming Weight

It is defined as the count of number of non-zero components in the code vector „C‟.
For example, if C1=1010101 and C2=1010001 are the valid code vectors then Hamming
weight of the C1 and C2 will be of 4 and 3 respectively.
4.10.2 Hamming Distance

It is defined as the number of mis-matched components between the code vector C1
and C2. For example, if C1=10101 and C2=10100 are the valid code vectors then
Hamming distance of the C1 and C2 will be 1.
It is noticed that, the values from 1st to 4th position is same in C1 and C2. But it differs
in 5th position of the code vector. Hence the Hamming distance between C1 and C2 is
said to be 1.
4.10.3 Hamming Bound

If an (n, k) linear block code is having capabilities of correcting „t‟ errors then the total
number of syndromes should not be less than the total number of all possible error
patterns and is given in equation (4.21).
………………….4.21
4.10.4 Minimum Distance [dmin]
It is the shortest hamming distance between any two code vectors in a code.
211
P=[ ]
Find the possible code vectors, Hamming Weights and parity check matrix. Also draw
the encoding circuit for the above codes.
Solution: For a (6, 3) linear block code, i.e., n=6 and k=3. Hence it generates 2k=23=8
message vectors. The code vectors are formed using equation (4.7).
[C] = [D] [G]

[G] = [ Ik P]
where „Ik‟ denotes identity matrix for the different values of „k‟. Since k=3, then the
[G] = [ I3 P]
[G]=[ ]
[C]=[d1, d2, d3] [ ]
the 2k message vectors.
212

Message Vector
Code Vector [C] for (6,3) linear block code
[D]
C4=d1 + C5=d1 + C6=d2 + Hamming
d1 d2 d3 C1=d1 C2=d2 C3=d3
d3 d2 d3 Weight
0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 1 3
0 1 0 0 1 0 0 1 1 3
0 1 1 0 1 1 1 1 0 4
1 0 0 1 0 0 1 1 0 3
1 0 1 1 0 1 0 1 1 4
1 1 0 1 1 0 1 0 1 4
1 1 1 1 1 1 0 0 0 3
The hamming distance for 000000 and 001101 is 3. For 001101 and 010011 it is
recorded as 4. In case of 010011 and 011110 the hamming distance is 3. The distance
between 011110 and 100110 is 3. In case of 100110 and 101011 its distance is 3. For the
vectors 101011 and 110101 its hamming distance is recorded as 4 and finally the
hamming distance between 110101 and 111000 is 3. To obtain parity check matrix, we
need to have the value of Parity matrix and Identity matrix (In-k=3). The parity check
matrix is given by
[H] = [In-k PT] = [I3 PT]
Therefore
[H] = [ ]
By observing the components of [H], it is noticed that minimum weight for the parity
check matrix [H] is dmin=3. Now for the possible code vector [C], the code vector bits
are [C1=d1, C2=d2, C3=d3, C4=d1 + d3, C5=d1 + d2, C6=d2 + d3]. It is required to draw the
encoding circuit for (6, 3) linear block code. Since k=3, it is necessary to consider three
shift register to shift the message bits. Also requires (n-k)=(6-3)=3 modulo-2 adders to
construct the circuit and is shown in Fig 4.9.
213
Fig. 4.9: Encoding Circuit for (6, 3) linear block code
Example 4.6: Consider d1, d2, d3 and d4 are the message bits. For a parity check bits of
(7, 4) hamming code generated by
C5=d1+d3+d4
C6=d1+d2+d4
C7=d2+d3+d4
(i) Find the generator matrix [G] and Parity check matrix [H] for the code.
(ii) Show that [G] [H]T=0.
(iii) Also find the Hamming weights, distances and minimum weight of this code.
Solution: (i) For a (7, 4) linear block code, i.e., n=7 and k=4. Hence it generates 2k =16
message vectors. It is given that,
[C] = [d1, d2, d3, d4, d1+d3+d4, d1+d2+d4, d2+d3+d4]
[C] = [d1, d2, d3, d4] [ ]
………………….4.22
214
The valid code vector is given by

[C] = [D] [G]
By comparing the above equations (4.7) and (4.21), the generator matrix is obtained
and is given by
[G]=[ ]
The generator matrix is in the form of
[G] = [ Ik P]= [ I4 P]
where „Ik‟ denotes identity matrix for the different values of „k=4‟. By knowing the
values of transpose of Parity matrix (P) and Identity matrix (In-k=I3) to generate the
Parity Check Matrix is obtained by
[H] = [PT In-k] = [PT I3]
Therefore
[H] =[ ]
(ii) Considering the values of generator matrix [G] and the transpose of parity check
matrix [H] in part (i). It is required to find
[G] [H]T=[ ]
[ ]
215
[G] [H]T=[ ]
Therefore
[G] [H]T=0
(iii) Table 4.8 provides the detailed information of systematic (7, 4) linear block code
for the 2k=24=16 message vectors.
Message Vector [D]

Code Vector [C] Hamming Weight
d1 d2 d3 d4
0 0 0 0 0000000 0
0 0 0 1 0001111 4
0 0 1 0 0010101 3
0 0 1 1 0011010 3
0 1 0 0 0100011 3
0 1 0 1 0101100 3
0 1 1 0 0110110 4
0 1 1 1 0111001 4
1 0 0 0 1000110 3
1 0 0 1 1001001 3
1 0 1 0 1010011 4
1 0 1 1 1011100 4
1 1 0 0 1100101 4
1 1 0 1 1101010 4
1 1 1 0 1110000 3
1 1 1 1 1111111 7
The hamming distance for 0000000 and 0001111 is 4. For the 0001111 and 0010101 it
results 3. Similarly we need to generate for all the pair of code vectors. It is noticed
216
that the minimum hamming distance is 3 which is equal to minimum hamming

weight observed in Table 4.8.
4.11 Single Error Correcting Hamming Codes

Some of the parameters such as code length, number of message bits, number of
parity check bits and error correcting capability are defined for correcting single error
pattern of (n, k) hamming code. Table 4.9 provides the details of single error
correcting (n, k) hamming code.
Table 4.9 Details of single error correcting (n, k) hamming code.
Parameters Definitions
Code length n 2n-k -1
Number of message bits k - ( )
Number of parity check bits (n-k)
Error correcting capability t=
Example 4.7: For a linear block code the minimum distance is three and a message
block size of eight bits. Find the generator matrix [G] and Parity check matrix [H].
Solution: From the Table 4.9, the code length „n‟ is given by
n 2n-k -1
Therefore
n 2n-8 -1
Apply trial and error method such that least integer value of „n‟ must satisfy
inequality. Suppose for n=11, then
11 211-8 -1=10
The value of 10 is less than compared with 11. Hence it is not possible to choose 11.
217
Suppose for n=12, then

12 212-8 -1=15
The value of 15 is greater than compared with 12. Hence it is possible to choose 12.
Therefore such an arrangement is composed of (12, 8) code. The transpose of parity
check matrix [H] is given by
( )
[H]T=[ ]=[ ]=
[ ]
Therefore the parity check matrix is obtained by
[H] = [ ]
The generator matrix [G] is given by
[G]= [Ik Pkx(n-k)] = [I8 P8x4]
[G]=
[ ]
218
Example 4.8: For a linear block code the minimum distance is three and a message
block size of three bits. Design the (n, k) hamming code. Also correct the error for
received vector R=[110111].
Solution: From the Table 4.9, the code length „n‟ is given by
n 2n-k -1
Therefore
n 2n-3 -1
Apply trial and error method such that least integer value of „n‟ must satisfy
inequality. Suppose for n=5, then
5 25-3 -1=3
The value of 3 is less than compared with 5. Hence it is not possible to choose 5.
Suppose for n=6, then
6 26-3 -1=7
The value of 7 is greater than compared with 6. Hence it is possible to choose 6.
Therefore such an arrangement is composed of (6, 3) code. The transpose of parity
check matrix [H] is given by
( )
[H]T=[ ]=[ ]=
[ ]
………………….4.23
Several conditions to be followed to form the parity matrix with the matrix order of
three rows and three columns. Some of the requirements are listed below
(i) The transpose of parity check matrix [H]T should not contain a row of zeros.
(ii) No two rows of [H]T must be same.
Since we have 2n-k=26-3=8 combinations of three bit numbers. To choose the

components for parity matrix,
219
[P]=[ ]
………………….4.24
Substitution of equation (4.24) in equation (4.23) to obtain transpose of parity check
matrix [H]T
[H]T=
[ ]
Therefore the parity check matrix (Hamming matrix) is obtained by
[H] = [ ]
By comparing the components of [H], the minimum hamming distance (dmin=3). The
generator matrix [G] is given by
[G]= [Ik Pkx(n-k)] = [I3 P3x3]
[G]= [ ]
The valid code vectors are found using equation (4.7)
[C] = [D] [G]
[C] = [d1, d2, d3, d2+d3, d1+d3, d1+d2]
Table 4.10 provides the details of the valid code vector for 2k=23=8 message vectors.
220
Message Vector [D]

Code Vector [C] Hamming Weight
d1 d2 d3
0 0 0 000000 0
0 0 1 001110 3
0 1 0 010101 3
0 1 1 011011 4
1 0 0 100011 3
1 0 1 101101 4
1 1 0 110110 4
1 1 1 111000 3
Since it is recorded that the value of dmin=3 from Table 4.10. Therefore the error
correcting capability of code is
t= = =1
Therefore (6, 3) hamming code formed is considered as single error correcting code.
Let the received vector be given as R=[11011]. Then the syndrome is calculated by
[S]=[R] [H]T
[S]= [1 1 0 1 1 1]
[ ]
[S]= [0 0 1]
Since the syndrome vector lies in the sixth position of the transpose of parity check
matrix [H]T. where the error pattern is represented by E= [0 0 0 0 0 1], since it lies in
the 6th position of [H]T. So the corrected code vector is obtained as
221
[C] = [R] + [E]
[C] = [110111] + [0 0 0 0 0 1]
[C] = [1 1 0 1 1 0]
The code vector [C] = [1 1 0 1 1 0] is a valid code vector for the received vector [1 1 0 1
1 1] that is tabulated in 4.10 for the message vector D = [1 1 0].
4.12 Table lookup Decoding using the standard array

The array is said to be a standard array, if it satisfies the following properties
(i) Each component in the standard array is same.
(ii) First n-tuple of each co-set is considered as co-set leader.
Based on the properties of the standard array, the decoding procedure is as follows
(i) The syndrome [S] = [R] [H]T is generated for the received vector.
(ii) The co-set leader Ei is located with [S] = [E] [H]T.
(iii) The corrected code vector [C] is obtained from [C] = [R] + [E]
The general decoding circuit for (n, k) linear code is described in Fig 4.10., by
considering the truth table of n-switching functions.
e1 = f1 (s1, s2,…., sn-k)
e2 = f2 (s1, s2,…., sn-k)
en = fn (s1, s2,…., sn-k)
Where s1, s2,…., sn-k are syndrome digits and e1, e2, …., en are the predicted error digits
of different co-set leaders.
222
Fig. 4.10: General decoding circuit for (n, k) linear code.
Example 4.9: Contruct the standard array for (5, 2) single error correcting code with
the generator matrix [G]
[G]=⌈ ⌉
Solution: We know that the standard array is capable to correct only the single error
pattern. It is given that n=5 and k=2. Therefore 2k=22=4 input combinations of binary
bits such as [D]=00, 01, 10 and 11. And 2n=25=32 numbers of codes are generated in
the array. Now multiply the each input of [D] with the generator matrix [G] to
generate codewords using equation (4.7).
For [D] = [0 0], the code vector [C]= [0 0] ⌈ ⌉= [0 0 0 0 0]
Now to construct the standard array, let us note down all the generated code words
along the first row in the given Table 4.11. The least weighted (less number of 1‟s)
223
numbers are considered in the first column of the second row (0 0 0 0 1) and perform
addition with all the contents of the first row. Similarly for the next least weighted
number say (0 0 0 1 0) and perform addition with all the contents of the first row.
Finally the procedure is continued till the least weighted numbers are completed.
Table 4.11: Standard array for (5, 2) single error correcting code
Standard array (5, 2)

00000 01101 10111 11010
00001 01100 10110 11011
00010 01111 10101 11000
00100 01001 10011 11110
01000 00101 11111 10010
10000 11101 00111 01010
10100 11001 00011 01110
10001 11100 00110 01011
Using the above procedure, it is required to generate only 24 codes instead of 32

possible codewords. So we need to generate the remaining eight codes by choosing
the least weighted number which was not appeared in earlier calculations. Hence (1 0
1 0 0) and (1 0 0 0 1) are chosen in the 7th and 8th row of 1st column and perform
addition with all the contents of the first row to generate remaining eight codewords.
Finally 2n=25=32 number of codes are appeared in the standard array table.
Further it can be concluded that first 24 codes which has least weighted number
(number of one‟s is 1) contains only one error and it can be corrected. But for the
change of weighted number (number of one‟s is 2), the errors are not possible to
correct it and results failure in decoding procedure.
Example 4.10: Contruct the standard array for (4, 2) of single error correcting code.
Find the status of error detecting and error correcting capabilities of code and form a
syndrome table for correctable error patterns. Also draw the encoding and decoding
circuit.
224
Solution: For (4, 2) linear block code, we have n=4 and k=2. Therefore, it is required
to obtain the parity matrix [P] with the order 2x2 for generating codes and to
construct standard array.
To choose the elements for parity matrix [P]
[P]=[ ]
………………….4.25
Since k=2, 2k=22=4 input combinations of binary bits such as [D]=00, 01, 10 and 11. For
parity matrix, the binary values such as 01 and 10 cannot be considered as elements
since both the values are representing identity matrix. Similarly message 00 is
representing syndrome with no-error, so cannot be used as an element in [P]. Hence it
is noticed that either 01 or 10 are consider as an element along with the remaining
pair 11 to construct the parity matrix. Therefore the parity matrix [P] with the order of
2x2 is given in equation 4.26.
[P]=[ ]
………………….4.26
The generator matrix [G] is obtained by
[G]= [Ik : P]=[I2 : P]

Therefore
[G]=[ ]
The parity check matrix [P] is given by
[H]=[ PT: In-k]=[ PT: I2]

Therefore
[H]= [ ]
225
We know that the standard array is capable to correct only the single error pattern. It
is given that n=4 and k=2. Therefore 2k=22=4 input combinations of binary bits such as
[D]=00, 01, 10 and 11. And 2n=24=16 numbers of codes are generated in the array.
Now multiply the each input of [D] with the generator matrix [G] to generate
codewords using equation (4.7).
For [D] = [0 0], the code vector [C]= [0 0] ⌈ ⌉= [0 0 0 0]
Now to construct the standard array, let us note down all the generated code words
along the first row in the given Table 4.12. The least weighted (less number of 1‟s)
numbers are considered in the first column of the second row (0 0 01) and perform
addition with all the contents of the first row. Similarly for the next least weighted
number say (0 0 1 0) and perform addition with all the contents of the first row.
Finally the procedure is continued till the least weighted numbers are completed.
Table 4.12: Standard array for (4, 2) single error correcting code

0000 0110 1011 1101
0001 0111 1010 1100
0010 0100 1001 1111
0100 0010 1111 1001
1000 1110 0011 0101
Using the above procedure, it is required to generate 20 codes instead of 16 possible

codewords. So we need to neglect the last four codes (1000, 1110, 0011 and 0101).
226
Finally 2n=24=16 number of codes are appeared in the standard array table. Further it
can be concluded that first 16 codes which has least weighted number (number of
one‟s is 1) contains only one error and it can be corrected. For the message vector
D=[00, 01, 10, 11], the code vectors along with the hamming weight [H w] is tabulated
in Table 4.13.
Message Vectors Code Vectors Hamming Weight
[D] [C] [Hw]
00 0000 0
01 0110 2
10 1011 3
11 1101 3
From the Table 4.13, dmin=min [Hw]=2. Error detecting capability dmin-1=2-1=1.
Therefore error correcting capability
t= = =
Therefore a (4, 2) linear block code is incapable of correcting any error. The syndrome
for correctable error pattern is given in Table 4.14.
Table 4.14: Syndrome for correctable error pattern.

0000 0110 1011 1101
0001 0111 1010 1100
0010 0100 1001 1111
0100 0010 1111 1001
The encoding circuit for (4, 2) linear block code is obtained by
[C]=[d1, d2]=[ ]
Therefore
[C]=[d1, d2, (d1+d2), d1]
227
Fig. 4.11: Encoding circuit for (4, 2) linear block code.
The syndrome vector [S] is obtained by
[S]= [R] [H]T=[r1 r2 r3 r4][ ]
[S] =[(r1+r2+r3), (r1+r4)]
Syndrome Calculation circuit can be given as shown in Fig. 4.12.
Fig. 4.12: Syndrome Calculation Circuit for (4, 2) linear block code.
228
THINGS TO BE
REMEMBER
1. The term Error Control Coding 8. Hamming distance is obtained as

approaches provides systematic the number of mis-matched
addition of redundant symbols components between the codes C1
and C2 and its distance is 1.
2. The codewords of a systematic
C1=1 0 1 0 1
linear block codes is given by
C2=1 0 1 0 0
[C]=[D][G]
3. Generator matrix [G] is obtained by 9. Hamming Bound is obtained by
[G]=[Ik:P] or [G]=[P:Ik] ∑
4. The Parity Check matrix [H] is

10. The number of information bits for
obtained by
single error correcting (n, k)
[H]=[PT:In-k]
hamming code is
k - ( )
5. The product of generator matrix [G]
and transpose of parity check matrix
11. The Code length for single error
[H] is zero.
correcting (n, k) hamming code is
[G][H]T=0
n 2n-k -1
6. The syndrome is calculated using
12. The error correcting capability for

[S]= [R][H]T
single error correcting (n, k)
hamming code is
7. Hamming Weight is defined as the
count of number of non-zero
components in the code vector „C‟. t=
229
DESCRIPTIVE
QUESTIONS
4. Define the following terms with
1. Explain briefly the communication their notations
system employing error control (a) Hamming Weight
coding with necessary block (b) Hamming Distance
diagram. (c) Minimum Distance
(d) Hamming Bound
2. If „C‟ is a valid code vector, then
Prove that [C][H]T=0 where [H]T is 5. Design (n, k) hamming code with a
the transpose of parity check minimum distance of dmin=3 and a
matrix [H]. message length of 4 bits.
3. For a systematic (7, 4) linear block 6. Design a single error correcting

code, the parity matrix „P‟ is given code with a message block size of
by 11 and show that by an example
that it can correct single error.
[P]=[ ]
7. Consider a (6, 3) linear code whose
generator matrix is
(i) Find all possible valid code
vectors.
[G]=[ ]
(ii) Draw the corresponding
encoding circuit.
(iii) A single error has occurred in (i) Find all possible valid code
each of these received vectors. vectors.
Detect and correct those errors. (ii) Find all the hamming weights
(a) RA=[0111110] and distances.
(b) RB=[1011100] (iii) Find minimum weight parity
(c) RC=[1010000] check matrix.
(iv) Also draw syndrome (iv) Draw the corresponding
calculation circuit. encoding circuit.
230
8. For the systematic (7, 4) linear

block code. Calculate
(i) The Valid code vectors for the
parity matrix [P] given below.
[P]=[ ]
(ii) Construct the standard array.

(iii) Find the status of error
detecting and error correcting
capabilities of code. Draw the
corresponding encoding and
decoding circuit.
231
MULTIPLE CHOICE
QUESTIONS
1. In the repetition code, how many 5. Which among the following

information bits are present in represents the code in which
addition to (n-1) parity bits? codewords consists of message bits
a. One and parity bits separately?
b. Two a. Block Codes
c. Four b. Systematic Codes
d. Eight c. Code Rate
d. Hamming Distance
2. For a (6, 3) block code with dmin=3.
How many errors can be corrected? 6. Redundancy in Communication
a. 0 Systems
b. 1 a. Reduces efficiency of the
c. 2 communication
d. 3 b. helps to detect the errors
c. helps to correct the errors
3. The minimum distance of linear d. All the above
block code (dmin) is equal to
minimum number of rows and
7. The main purpose of coding is to
columns of HT, whose …….is equal
improve the
to zero vector.
a. Signal to noise ratio
a. sum
b. Bit error rate
b. difference
c. Selectivity
c. product
d. Linearity
d. division
4. In a linear code, the minimum 8. The Hamming Code is an

hamming distance between any two a. Error detecting code system
code words is_____ minimum b. Error correcting code system
weight of any nonzero code word. c. Error analyzes code system
a. Less than d. None of the above
b. Greater than
c. Equal to
d. None of the above
232
9. One of the major characteristics of 10. Which among the following are the
linear block codes conventional essential conditions for a good error
codes is control coding techniques?
a. Both of them have memory a. Faster coding & decoding
b. none of them have memory methods
c. Only linear block codes have b. Better error correcting capability
memory c. Maximum transfer of information
d. Only conventional codes have in bits/sec
memory. d. All of the above
Answers
Q. No 1 2 3 4 5
Options a b a c b
Q. No 6 7 8 9 10
Options d b b d d
References
[1] S. Lin and D. J. Costello, Error Control Coding, 2nd Edition, Prentice Hall, 2004.
[2] R. E. Blahut, Algebraic Codes for Data Transmission, Cambridge University Press,
2002.
[3] T. K. Moon, Error Correction Coding: Mathematical Methods and Algorithms, Wiley,
2005.
[4] R. H. Morelos-Zaragoza, The Art of Error Correcting Coding, Wiley and sons, 2006.
New York; 1989.
Delhi; 2008.
and Sons: 1989.
233
Work Space
234
Chapter 5
Binary Cyclic Codes

Syllabus: Binary Cyclic Codes: Algebraic Structure of Cyclic Codes, Encoding using an
(n-k) Bit Shift register, Syndrome Calculation, Error Detection and Correction.
Contents
5. Binary Cyclic Codes Page. No
Learning Objectives
5.1 Introduction...........................................................................................................................231
5.2 Algebraic structure of cyclic codes.....................................................................................231
5.3 Properties of cyclic codes.....................................................................................................234
5.4 Encoding using (n-k) bit shift registers..............................................................................240
5.5 Syndrome calculation...........................................................................................................248
5.6 Error detection and correction.............................................................................................249
References
235
Information: the negative
reciprocal value of
probability.
Claude. E. Shannon
236
5 BINARY CYCLIC CODES
Learning Objectives

Understand the need for cyclic codes over linear block codes.
Analyze cyclic codes and investigate the relationship between minimum distance and error
correcting capabilities.
Design an encoder and syndrome calculator circuit for (n-k) shift registers.
5.1 Introduction
Binary Cyclic Codes (BCC) forms a subclass of linear block codes which was
discussed in previous chapter 4. The BCC has several advantages linear block codes
such as implementation of encoding and syndrome calculation circuits is easy with
simple shift register, basic gates with feedback registers. Also, cyclic codes provides
better algebraic structures which leads to detect and correct the errors in an efficient
manner. This chapter deals with the development of an efficient polynomial
representation for cyclic codes and makes use of codes for encoding and decoding
purpose.
5.2 Algebraic Structure of Cyclic Codes

The term cyclic code is defined as if code ‘C’ generated from the (n, k) linear block
code is said to be cyclic, if every cyclic shifts of the code is also belongs to code ‘C’.
For example if Ca=010101 be a code vector of ‘C’. If C2= 101010 is also code vector of
‘C’ and it is said to cyclic code. In general, let the n-tuple be represented by
V= (V0, V1, V2,……Vn-2)

………………….5.1
The polynomial representation of equation (5.1) is given by
V(x)= V0 + V1x V2 x2,……Vn-1 xn-1
237
………………….5.2
If ‘V’ belongs to cyclic code then
V(1)=(Vn-1, V0, V1, V2 ….Vn-2)
V(2)=(Vn-2, Vn-1, V0, V1, V2 ….Vn-3)
V(i)=(Vn-i, Vn-i+1,V0, V1, V2 ….Vn-3)

………………….5.3
Further the polynomial representation of equation (5.3) is
V(1)x=(Vn-1+V0x+V1x2+ ….+Vn-2 xn-1)
V(2)x=(Vn-2+Vn-1x+V0x2+ ….+Vn-3 xn-1)
V(i)x=(Vn-i+Vn-i+1x+Vn-i+2x2+….Vn-i-1xn-1)
………………….5.4
Where V0, V1, V2…Vn-1 belongs to binary and performs modulo 2 arithmetic such as
addition and multiplication. Table 5.1 provides the operation of modulo 2 addition
and multiplication procedure.
Table 5.1: Modulo-2 Addition and Multiplication
Modulo-2 Addition Modulo-2 Multiplication
0+0 0 0.0 0
0+1 1 0.1 0
1+0 1 1.0 0
1+1 0 1.1 1
Example 5.1: Perform multiplication for the polynomials f1(x)=(1+x) and

f2(x)=(1+x+x2) using modulo-2 algebra.
Solution: The product of two functions obtained as
238
Binary Cyclic Codes
f1(x) f2(x)=(1+x)(1+x+x2)
=1+x+ x2+x+ x2+x3
=1+ x3+ x2+ x2+x+x
=1+ x3+ x2(1+1)+x(1+1)
=1+ x3+0+0
Therefore
f1(x).f2(x)=1+ x3
Example 5.2: Show that the function (x4+x3+1)2= x8+x6+1 using modulo-2 algebra
Solution: Consider (x4+x3+1)2=(x4+x3+1) (x4+x3+1)

= x8+x7+x4+x7+x6+x3+x4+x3+1
=1+x8+x7+x7+x6+x4+x4+x3+x3
=1+x8+x7(1+1)+x6+x4(1+1)+x3(1+1)
=1+x8+x6
Therefore
(x4+x3+1)2= x8+x6+1
Example 5.3: Divide the polynomial f2(x)=(x8+x5+x) and f1(x)=(x2+x+1) using

modulo-2 algebra.
Solution: For the operation of division, ‘Q(x)’ and ‘R(x)’ denotes quotient and
remainder polynomial respectively.
x2+x+1) x8+x5+x ( x6+x5……….Q(x)
x8+x7+x6
x7+x6+x5+x
239
x7+x6+x5
x…………R(x)
5.3 Properties of Cyclic Codes:

There are five important properties of the cyclic codes are discussed in this section
with the consideration of generator polynomial, parity check polynomial etc.
First Property:
The generator polynomial g(x) generated for the (n, k) cyclic code with a degree of (n-
k) is unique.
g(x)=g0 + g1x + g2x2+…..+gn-k xn-k
………………….5.5
Second Property:
The generator polynomial g(x)=g0 + g1x + g2x2+…..+gn-k xn-k generated for the (n, k)
cyclic code is a factor of xn+1.
xn+1=g(x) h(x)
………………….5.6
Where ‘h(x)’ is called Parity check polynomial.
Third Property:
A (n, k) cyclic code is generated, if g(x) is a polynomial of degree (n-k) and is a factor
of xn+1.
Fourth Property:
The code vector polynomial V(x) can be generated by
V(x) =D(x) g(x)

………………….5.7
Where D(x) = d0+d1x+d2x2+….+ dk-1xk-1
is a message vector polynomial of degree ‘k’.
This approach generates non-systematic cyclic codes.
Fifth Property:
240
Binary Cyclic Codes
A systematic cyclic code is generated and obtains a remainder polynomial from the
division of Xn-k D(x) by g(x).
Fig. 5.1: Formation of systematic cyclic code

The arrangement of systematic cyclic codes for n-bit code vector is shown in fig 5.1.
Example 5.4: Perform non-systematic cyclic code operation to generate a code for the
(6, 3) single error correcting code with D(x)=d0+d1x+d2x2 and g(x)=1+x2.
Solution: The non-systematic cyclic code-vectors can be obtained using equation

(5.7)
V(x) =D(x) g(x)
Now let us consider the message vector D=[d0, d1, d2] is [0 0 1]. Then its polynomial is
D(x)= d0+d1x+d2x2
D(x)= (0)+(0)x+(1)x2
D(x)= x2
Substituting D(x) in equation (5.7)
V(x) = (x2) (1+x2)
V(x)= x2+ x4
The code vector for the message input [0 0 1] is
[V]=[0 0 1 0 1 0]
D(x)= d0+d1x+d2x2
241
D(x)= (0)+(1)x+(0)x2
D(x)= x
V(x) = (x) (1+x2)
V(x)= x+ x3
[V]=[0 1 0 1 0 0]
D(x)= d0+d1x+d2x2
D(x)= (0)+(1)x+(1)x2
D(x)= x+ x2
V(x) = (x+ x2) (1+x2)
V(x)= x+ x3+ x2+ x4
V(x)= x+ x2+ x3+ x4
[V]=[0 1 1 1 1 0]
D(x)= d0+d1x+d2x2
D(x)= (1)+(0)x+(0)x2
D(x)= 1
242
Binary Cyclic Codes
V(x) = 1 (1+x2)
V(x)= 1+ x2
V(x)= 1+ x2
[V]=[1 0 1 0 0 0]
D(x)= d0+d1x+d2x2
D(x)= (1)+(0)x+(1)x2
D(x)= 1+x2
V(x) = (1+x2) (1+x2)
V(x)= 1+x2+ x2+ x4
V(x)= 1+ x4
[V]=[1 0 0 0 1 0]
D(x)= d0+d1x+d2x2
D(x)= (1)+(1)x+(0)x2
D(x)= 1+x
V(x) = (1+x) (1+x2)
V(x)= 1+x2+x+ x3
243
V(x)= 1+x+x2+x3
[V]=[1 1 1 1 0 0]
D(x)= d0+d1x+d2x2
D(x)= (1)+(1)x+(1)x2
D(x)= 1+x+x2
V(x) = (1+x+x2) (1+x2)
V(x)= 1+x2+x+ x3+ x2+x4
V(x)= 1+x+x3+x4
[V]=[1 1 0 1 1 0]
Table 5.2: Non systematic code vectors
Message (D) Code Vector (V)

000 000000
001 001010
010 010100
011 011110
100 101000
101 100010
110 111100
111 110110
244
Binary Cyclic Codes
Example 5.5: Perform systematic cyclic code operation to generate a code for the
(6, 3) single error correcting code with D(x)=d0+d1x+d2x2 with [D]=[0 1 1] and [1 0 0]
and g(x)= x2+1.
Solution: For systematic cyclic code operation, we have
R(x)=
………………….5.8
For the message vector [D]=[0 1 1], the message vector polynomial will be
D(x)=x+x2
Therefore
=( )(x+x2)
=x5+x4
Substituting the values of and g(x) in equation (5.8).
R(x)=
x2+1) x5+x4 (x3+x2+x+1 ……….Q(x)

x5+x3
x4+x3
x4+x2
x3+x2
x3+x
x2+x
x2+1
x+1 …………R(x)
Therefore the remainder polynomial is
R(x)=1+x=1+(1)x+(0)x2
245
Hence the remainder vector [R]= [1 1 0]
Therefore the code vector [V] for the message vector [0 1 1] is
[1 1 0 0 1 1 ]
For the message vector [D]=[1 0 0], the message vector polynomial will be
D(x)=1
Therefore
=( )
=x3
Substituting the values of and g(x) in equation (5.8).
R(x)=
x2+1) x3 (x……….Q(x)
x3+x
x…………R(x)
Therefore the remainder polynomial is
R(x)=x=0+(1)x+(0)x2
Hence the remainder vector [R] = [0 1 0]
Therefore the code vector [V] for the message vector [1 0 0] is
[0 1 0 1 0 0]
Table 5.3: Systematic code vectors
Message (D) Code Vector (V)

011 110011
100 010100
246
Binary Cyclic Codes
5.4 Encoding using (n-k) Bit Shift Registers:

As, we have discussed in the previous section, the remainder polynomial R(x) is
obtained by dividing xn-k D(x) from generator polynomial g(x). This process can be
accomplished using dividing circuit consisting of feedback shift register, modulo-2
adders, and basic and gate as shown in fig 5.2.
Fig. 5.2: Encoding circuit for (n, k) cyclic code.

Where
R….... Flip flop that makes up shift registers
….. Modulo-2 adder
gi…… if gi=0, its open and acts as a closed path for gi=1
And gate.
The encoding circuit using (n-k) shift register operates during two cases, when the
gate is turned ON and turned OFF respectively.
During gate ON, it is noticed that the switch is in position 1 and the message
digits (d0, d1,…..,dk-1) starting from ‘dk-1’ are shifted into the registers towards
247
the channel. After completion of shifting procedure of ‘k’ message digits, now
the register holds the components of parity check bits (R0, R1,….., Rn-k-1).
During gate OFF, it is noticed that the switch is in position 2 and the contents
of shift register are moved to the channel. Hence, the code vector (R0, R1,….,
Rn-k-1, d0, d1,…..,dk-1) is generated and further fed to the communication
channel.
Example 5.6: Design an encoder for the (8, 5) binary cyclic code generated by
g(x)=1+x2+x3 and verify its operation using the message vectors [D]=[1 1 0 0 1].
Solution: To design encoder for BCC, from equation (5.5) we have in general
generator polynomial is given by
g(x)=g0 + g1x + g2x2+…..+gn-k xn-k
For (8, 5) cyclic code we know that,

g(x)=1+x2+x3
By comparing given g(x) with equation (5.5) we have
g0=1, g1=0, g2=1, g3=1
The number of flip-flops required is obtained by using Rn-k-1=R8-5-1=R2 (i.e., R0, R1 and
R2). Also the design requires two module- 2 adders.
248
Binary Cyclic Codes
Fig. 5.3: Encoding circuit for (8, 5) cyclic code.
For the message D= [1 1 0 0 1], the contents of shift registers are tabulated in Table 5.4
Table 5.4: Contents of shift registers for [D]=[1 1 0 0 1]
No. of Shifts Input [D] Shift register Contents Remainder Bits

Initially Switch ‘S’ is in R
R0 R1 R2
position 1, indicates Gate is
turned ON 0 0 0 -
1 1 1 0 1 -
2 0 1 1 1 -
3 0 1 1 0 -
4 1 1 1 0 -
5 1 1 1 0 -
Now, Switch ‘S’ moves to
turned OFF
6 x 0 1 1 0
7 x 0 0 1 1
8 x 0 0 0 1
249
Example 5.7: Design an encoder for the (31, 21) binary cyclic code generated by
g (x)=1+x3+x5 +x6+x8+x9+x10.
g (x)=g0 + g1x + g2x2+…..+gn-k xn-k
For (31, 21) cyclic code we know that,

g (x)=1+x3+x5 +x6+x8+x9+x10.
g0=1, g1=0, g2=0, g3=1, g4=0, g5=1, g6=1, g7=0, g8=1, g9=1, g10=1
The number of flip-flops required is obtained by using Rn-k-1=R31-21-1=R9 (i.e., R0, R1, R2,
R3, R4, R5, R6, R7, R8 and R9). Also the design requires six module-2 adders.
Example 5.8: Find the code word in systematic form for the message [D] = [1010101],
250
Binary Cyclic Codes
where the generator polynomial of a (15, 7) cyclic code is g(x) = 1+ x4 + x6 + x7 + x8.

Also verify the results obtained using direct hand calculation.
g (x)=g0 + g1x + g2x2+…..+gn-k xn-k
For (15, 7) cyclic code we know that

g(x) = 1+ x4 + x6 + x7 + x8
g0=1, g1=0, g2=0, g3=0, g4=1, g5=0, g6=1, g7=1, g8=1.
The number of flip-flops required is obtained by using Rn-k-1=R15-7-1=R7 (i.e., R0, R1, R2,
R3, R4, R5, R6 and R7). Also the design requires four module-2 adders.

For the message [D] = [1010101], the contents of shift registers are tabulated in Table
5.5
Table 5.5: Contents of shift registers for [D]=[1010101]
No. of Shifts Input [D] Shift register Contents Remainder Bits

Initially Switch ‘S’ is in R
R0 R1 R2 R3 R4 R5 R6 R7
0 0 0 0 0 0 0 0 -
turned ON
251
1 1 1 0 0 0 1 0 1 1 -
2 0 1 1 0 0 1 1 1 0 -
3 1 1 1 1 0 1 1 0 0 -
4 0 0 1 1 1 0 1 1 0 -
5 1 1 0 1 1 0 0 0 0 -
6 0 0 1 0 1 1 0 0 0 -
7 1 1 0 1 0 0 1 1 1 -
Now Switch ‘S’ moves to
turned OFF
8 X 0 1 0 1 0 0 1 1 1
9 X 0 0 1 0 1 0 0 1 1
10 X 0 0 0 1 0 1 0 0 1
11 X 0 0 0 0 1 0 1 0 0
12 X 0 0 0 0 0 1 0 1 0
13 X 0 0 0 0 0 0 1 0 1
14 X 0 0 0 0 0 0 0 1 0
15 X 0 0 0 0 0 0 0 0 1
From the Table 5.5, the remainder polynomial coefficients are 1 0 1 0 0 1 1. For
systematic code the code vector for the message vector [D] = [1 0 1 0 1 0 1] is given by
[V] = [1 0 1 0 0 1 1 1 0 1 0 1 0 1]
To verify the code vector using direct hand calculation: From the equation (5.8) we
have,
R(x) = =
=x8(1+ x2+ x4+ x6) = x8+ x10+ x12+ x14
x8 + x7 + x6 + x4 + 1) x14+ x12+ x10+ x8 (x3+x2+x+1 ……….Q(x)

x14+ x13+ x12+ x10+ x6
x13+ x8+ x6
x13+ x12+ x11+ x9+ x5
x12+ x11+ x9+ x8+ x6+ x5
252
Binary Cyclic Codes
x12+ x11+ x10+ x8+ x4

x10+ x9+ x8+ x5+ x4
x10+ x9+ x8+ x6+ x2
x8+ x5+ x4+ x2
x8+ x7+ x6+ x4+ 1
R(x)……. x7+ x6+ x5+ x2+ 1
Hence, the result code vector generated by shift register method and direct calculation
method are same.
Example 5.9: For a linear Hamming code described by generator polynomial

g(x)=1+x+x3. Find the generator matrix [G] and parity check matrix [H]. Also draw an
encoder circuit.
Solution: to generate generator matrix [G] it is required to obtain the elements

of the matrix as below.
g(x)=1+x+x3 = [1101000]
x g(x)=x+x2+x4 =[0110100]
x2g(x)= x2+ x3+x5 = [0011010]
x3g(x)= x3+ x4+x6 = [0001101]
[G]= [ ]
………………….5.9
Equation (5.9) is not in standard form[ ], so following operations are need to
be carried out
R3 R3–R1 and R4 R4–R1
253
[G]= [ ]
R4 R4–R2
[G]= [ ]
So, the modified [G] is written as
[G]= [ ]
The parity check matrix is given by
[H] =[ ]=[ ]
[H] = [ ]
With g0=1, g1=1, g2=0 and g3=1, the encoder circuit is given by
Fig. 5.6: Encoding circuit for the message

254
Binary Cyclic Codes
5.5 Syndrome Calculation:

The term syndrome is used in the communication process for the decoder to check
whether the received vector is a valid code vector or invalid code vector.
If the syndrome is zero, then the received vector polynomial R(x) is divisible
by g(x) and results that the received vector is a valid code vector.
For the syndrome of non-zero, it is noticed that there are some errors in the
received vector. Hence we have
(x)+
………………….5.10
Where Z(x) is a received polynomial, Q(x) represents quotient polynomial and S(x)
denotes syndrome polynomial of degree (n-k-1). For the error pattern E(x) caused by
the communication channel which is given in equation (5.11).
[ ]
………………….5.11
The syndrome calculation circuit using (n, k) cyclic code as shown in Fig.5.7 operates
for the combinations of when the gate 2 is turned ON and gate 1 is turned OFF. In the
second case, when the gate 2 is turned OFF and gate 1 is turned ON.
First initialize the register and the content of the received vector [Z] are
entered into the shift register with the positions of gate 2 is ON and gate 1 is
OFF.
Once all the received vector is shifted into the register, then it is noticed that
the contents of the register will be syndrome. With the positions of gate 2 is
OFF and gate 1 is ON, the syndrome vector is moved out of the register.
255
Fig. 5.7: A (n-k) Syndrome calculation for (n, k) cyclic code.
S….... Flip flops
….. Modulo-2 adder
gi…… if gi=0, its open and acts as a closed path for gi=1
And gate.
5.6 Error Detection and Correction:

Many illustrations are discussed to generate a code vectors using linear block codes
and binary cyclic code approaches in the previous section. But it is clear that cyclic
codes are well suited in order to detect the errors than linear block codes while
generating the valid code vectors. So the general steps to correct the errors in cyclic
codes are tabulated in Table 5.6.
Table 5.6: General steps to correct errors in cyclic codes
256
Binary Cyclic Codes
Step1: The received vector is shifted into the memory storage device such as buffer register
and Syndrome register.
Step2: After placing the syndrome in the syndrome register, the contents of the syndrome
Register are read into the detector.
Step3: The first received bit is shifted out of the buffer and the syndrome register is shifted
right once. These two operations must be performed at the same time.
Step4: Now the new syndrome is used to check the second received digit. At the right most
stage of the buffer register, is an erroneous digit. If so, it is corrected.
Step5: The decoder operates continuously on the received vector digit by digit until all the
received vector is shifted out of the buffer.
At the end of decoding process in cyclic codes, the errors are corrected with respect to
the error pattern built into the detector. It is observed that after correcting the errors,
the content in the syndrome register results all zeros. Suppose if the content in the
syndrome register does not contain all zeros, then an uncorrectable error pattern has
been detected.
Example 5.10: If the received vector is [Z] = [1 0 1 1 1 0 0] for the generated

polynomial g(x)=1+x2+x3. Then design a syndrome calculation circuit for the (7, 4)
binary cyclic code and also correct single error in the received vector.
Solution: To design syndrome calculation circuit for BCC, from equation (5.5)
we have in general generator polynomial is given by
g (x)=g0 + g1x + g2x2+…..+gn-k xn-k
For (7, 4) cyclic code we know that
g(x)=1+x2+x3
g0=1, g1=0, g2=1, g3=1.
The number of flip-flops required is obtained by using Sn-k-1=S8-5-1=S2 (i.e., S0, S1 and
S2). Also the design requires two module-2 adders as shown in fig 5.8.
257
For the received vector [Z] = [1 0 1 1 1 0 0], the contents of shift registers are tabulated
in Table 5.7.
Table 5.7: Contents of shift registers for [Z] = [1 0 1 1 1 0 0]
No. of Input Shift register Contents

Comments
Shifts [Z] S0 S1 S2
Initially Gate 1 is
turned OFF Shift Register contents are
0 0 0
And Gate 2 is cleared
turned ON
1 0 0 0 0
2 0 0 0 0
3 1 1 0 0
4 1 1 1 0
5 1 1 1 1
6 0 1 1 0
7 1 1 1 1 Indicates Error
8 0 1 1 0
9 0 1 1 1
10 0 1 1 0
11 0 0 1 1
12 0 1 0 0 End of shifting operation
258
Binary Cyclic Codes
From the Table 5.7, it is noticed that 1 0 0 is appeared in the 12th shift, which indicates
error in the 5th bit from the right.
The received vector
So, the error vector for the above is considered as [E] = [0 0 1 0 0 0 0]. Therefore, the
corrected vector can be obtained as
[V] = [Z] + [E] = [1 0 1 1 1 0 0] + [0 0 1 0 0 0 0]
[V] = [1 0 0 1 1 0 0]
Thus the obtained vector [V] is a valid code vector.
259
THINGS TO BE
REMEMBER
1. The Binary Cyclic Codes (BCC) 6. A Non-Systematic Cyclic Code is

forms a subclass of linear block generated by
code. V(x) =D(x) g(x)
Where D(x) = d0+d1x+d2x2+….+dk-
1x
2. The term Cyclic Code is defined as k-1 is a message vector polynomial
if code ‘C’ generated from the (n, k) of degree ‘k’.

linear block code is said to be
cyclic, if every cyclic shifts of the 7. A Systematic Cyclic Code is
code is also belongs to code ‘C’. generated and obtains a remainder
polynomial from the division of Xn-
3. The generator polynomial g(x) k D(x) by g(x).
generated for the (n, k) cyclic code 8. For the syndrome of non-zero, it is
with a degree of (n-k) is unique. noticed that there are some errors
in the received vector. Hence we
g(x)=g0+g1x+g2x2+…..+gn-k xn-k have
(x)+
4. The generator polynomial
g(x)=g0+g1x+g2x +…..+gn-k
2 xn-k
Where Z(x) is a received
generated for the (n, k) cyclic code
polynomial, Q(x) represents
is a factor of xn+1.
quotient polynomial and S(x)
xn+1=g(x) h(x)
denotes syndrome polynomial of
degree (n-k-1).
5. A (n, k) cyclic code is generated, if
g(x) is a polynomial of degree (n-k)
9. For the Error pattern E(x) caused
and is a factor of xn+1.
by the communication channel
which is given by
[ ]
260
Binary Cyclic Codes
DESCRIPTIVE
QUESTIONS
1. Define Binary Cyclic code. Discuss (ii) Find code polynomial for
the features of Encoder and D(x)= 1+x2+x4 in systematic
Decoder used for Cyclic codes form.
using as (n-k) bit shift register. (iii) If v(x)=1+x4+x6+x8+x14 then
2. Explain briefly with relevant check whether it is valid code
examples polynomial or not.
(i) Hamming code 7. For the generator polynomial g(x)=
(ii) Cyclic codes 1+x+x3, find the generator matrix
3. List the distinct advantages of [G] and parity check matrix [H] of
cyclic codes over linear block (7, 4) cyclic codes.
codes. 8. Design an encoder for the (7, 4)
4. A generator polynomial of (7, 4) binary cyclic code generated by
cyclic code is G (P)=P3+P+1. Find g(x)=1+x+x3 and verify its
all the code vectors for the code in operation using the message
systematic and non-systematic vectors (1 0 0 0) and (1 1 0 0).
form. 9. For a (7, 4) cyclic code, the received
5. Consider (15, 11) cyclic code vector [Z]=[1 1 1 0 1 0 1] and the
generated by g(x)=1+x+x4. generator polynomial is g(x)=
(i) Device a feedback registers 1+x+x3. Draw the syndrome
encoder for this code. calculation circuit and correct the
(ii) Illustrate the encoding single error in the received vector.
procedure with message vector 10. For n 7, identify (n, k) values of
11001101011 by listing the the cyclic codes generated by the
states of the register. following generator polynomials.
6. A (15, 5) cyclic code has a generator (i) g(x)=1+x2+x3
polynomial (ii) g(x)=1+x+x2+x4
g(x)=1+x+x2+x4+x5+x8+x10 (iii) g(x)=1+x2+x3+x4
(i) Draw the block diagram of an
encoder and syndrome
calculator for this code.
261
MULTIPLE CHOICE
QUESTIONS
1. The cyclic codes are designed using 6. For the message 1010001101, the
a. Shift register with feedback CRC using divisor polynomial
b. Shift register without feedback x5+x4+x2+1 is
c. Flip-flops a. 01110
d. None of the above b. 01111
2. The received code contains error, if c. 11110
the syndrome vector is d. 01010
a. Zero 7. For the generation of cyclic codes,
b. non-zero the generator polynomial should be
c. Infinity the factor of
d. none of the above a. xn+1
3. In the dividing circuit, the parity b. xn-1
polynomial is obtained by the c. xn-k+1
a. Quotient d. xn-k-1
b. Reminder 8. For designing of (4, 1) cyclic
c. Dividend repetition code, what would be the
d. Divisor order of generator polynomial?
4. A cyclic code can be generated using a. 4
a. Generator polynomial b. 2
b. Generator matrix c. 1
c. Parity check matrix d. 3
d. None of the above 9. According to the linearity property,
5. The feedback shift register is called the…….of the two code words in a
as cyclic code is also a valid code
a. Multiplying word.
b. Shifting a. Multiplication
c. Feedback b. Difference
d. Dividing c. Division
d. Sum
262
Binary Cyclic Codes
10. While decoding the cyclic code, if

the received code word is similar as Answers
transmitted code word, then r(x) Q. No 1 2 3 4 5
mod g(x) is equal to Options a b b a d
a. Zero Q. No 6 7 8 9 10
b. Unity Options a a d d a
c. Infinity
d. None of the above.
References
[1] R. E. Blahut, Algebraic Codes for Data Transmission, Cambridge University Press,
2002.
[2] T. K. Moon, Error Correction Coding: Mathematical Methods and Algorithms, Wiley,
2005.
[3] R. H. Morelos-Zaragoza, The Art of Error Correcting Coding, Wiley and sons, 2006.
[4] W. Stallings, Cryptography and Network Security: Principles and Practice, 4th Edition,
Prentice Hall, 2006.
New York; 1989.
Delhi; 2008.
and Sons: 1989.
263
Work Space
264
Chapter 6
Some Important Cyclic Codes

Syllabus: Some important Cyclic codes such as Golay Codes and BCH Codes.
Contents
6. Some Important Cyclic Codes Page. No
Learning Objectives
6.1 Introduction...........................................................................................................................259
6.2 Galois field................................................................................................................. ............259
6.2.1 Galois field of 2n...........................................................................................................262
6.2.2 Representation of Polynomials..................................................................................263
6.2.3 Irreducible Polynomials..............................................................................................264
6.3 Bose-Chaudhuri-Hocquenqhem (BCH) codes..................................................................268
6.3.1 Generator Polynomial.................................................................................................268
6.3.2 Degree of BCH code....................................................................................................269
6.4 Reed Solomon codes.............................................................................................................271
6.4.1 Properties of Reed Solomon codes............................................................................271
6.4.2 Generator Polynomial.................................................................................................274
6.5 Golay codes........................................................................................................................ ...275
References
265
Information is the Resolution
Of Uncertainty
-Claude. E. Shannon
266
6 SOME IMPORTANT CYCLIC CODES
Learning Objectives

Define Bose-Chaudhuri-Hocquenghem(BCH)codes using generator and parity check
polynomials.
Present various decoding methods of RS, Golay and BCH codes.
Gain the ability to decode RS and BCH codes.
6.1 Introduction
In cryptography, Galois field GF(2n) is often used because it supports all the
arithmetic operations such as addition, subtraction, multiplication and division. For
GF(2n), the ‘n’ can hold the values of 1, 2, 3…..which has a particular value that plays
a prominent role in network security. For the sets of polynomials in GF(2n), a group of
polynomials of degree ‘n’ is defined as the modulus. Such polynomials are referred to
as irreducible polynomials. This chapter deals with the polynomial representation for
some important cyclic codes such as Golay codes and BCH codes.
6.2 Galois Field

Galois Field (GF) demonstrates that, for the field to be finite the number of elements
should be pn, where ‘p’ is a prime and ‘n’ is a positive integer. In other words
A Galois field, GF (pn) is a finite field with pn elements.

………………….6.1
Suppose if the value of ‘n’ is unity, then equation 6.1 can be written as
GF (p)
………………….6.2
267
Equation (6.2) indicates that, ‘p’ can be any prime number and the set become (Zn*)
with Zp={0, 1,…..,p-1} with two arithmetic operations such as addition and
multiplication are supported. Now let us consider an example 6.1 to describe the
Galois field.
Example 6.1: Consider a smallest value of p=2 for a Galois field GF(p) where ‘p’ is a
prime number with the set {0, 1}.
Solution: For a Galois field GF(p)= GF(2) with the set {0, 1} and two arithmetic
operators such as addition and multiplication. We have
GF(2)
{0, 1} + X
Fig. 6.1(a): Galois field operation with two operators
+ 0 1
0 0 1
1 1 0
Fig. 6.1(b): Addition
x 0 1
0 0 0
1 0 1
Fig. 6.1(c): Multiplication
a 0 1 a 0 1
-a 1 0
-a 1 0
Fig. 6.1(d): Additive Inverse Fig. 6.1(e): Multiplicative Inverse

Fig. 6.1(a),(b),(c),(d)(e): Galois Field for P=2.
268
Some Important Cyclic codes
Example 6.2: Obtain an GF(5) on the set ‘Z5’ where ‘5’ is a prime number with
addition and multiplication operators.
Solution: For a Galois field GF(p)= GF(5) with the set {0, 1, 2, 3, 4} and two arithmetic
operators such as addition and multiplication. We have
GF(5)
{0, 1, 2, 3, 4} + X
+ 0 1 2 3 4
0 0 1 2 3 4
1 1 2 3 4 0
2 2 3 4 0 1
3 3 4 0 1 2
4 4 0 1 2 3
Fig. 6.2(b): Addition
x 0 1 2 3 4
0 0 0 0 0 0
1 0 1 2 3 4
2 0 2 4 1 3
3 0 3 1 4 2
4 0 4 3 2 1
Fig. 6.2(c): Multiplication
a 0 1 2 3 4 a 0 1 2 3 4
-a 0 4 3 2 1 -a …. 1 3 2 4
Fig. 6.2(d): Additive Inverse Fig. 6.2(e): Multiplicative Inverse
Fig. 6.2(a),(b),(c),(d),(e): Galois Field for P=5.

269
6.2.1 Galois Field of (2n) or GF((2n)

When we come across the application of cryptography, Galois field GF(2n) is often
used because it supports all the arithmetic operations such as addition, subtraction,
multiplication and division. For GF(2n), the ‘n’ can hold the values of 1, 2, 3…..which
has a particular value that plays a prominent role in network security. Now let us
consider an example 6.3 for the description of GF(2n). The polynomials representing
n-bit words use two fields such as GF(2) and GF(2n).
Example 6.3: Let us define GF(22) field in which the set has four 2 bit words {00, 01,
10, 11}. Now we can redefine addition and multiplication for this code where all the
properties of these operations are satisfied.
Solution: For a Galois field GF(2n)= GF(22) with the set {00, 01, 10, 11} and two
arithmetic operators such as addition and multiplication. We have
GF(22)
{00, 01, 10, 11}

+ X
+ 00 01 10 11
00 00 01 10 11
01 01 00 11 10
10 10 11 00 01
11 11 10 01 00
Fig. 6.3(b): Addition Identity 00
270
x 00 01 10 11
00 00 00 00 00
01 00 01 10 11
10 00 10 11 01
11 00 11 01 10
Fig. 6.3(c): Multiplication Identity 01
Now let us consider the multiplication procedure where if 10 (i.e., 2) is multiplied

with 11 (i.e., 3) then it produces the result of 10 (i.e., 6 mod 4 = 2). But the value 01
(i.e., 1) is written in the multiplicative table 6.3(c). So it is required to generate
irreducible polynomial.
Now the polynomial representation for the set {00, 01, 10, 11} is {0, x, 1, 1+x}.The
polynomial after proper arrangement we have {0, 1, x, 1+x}. By using the polynomial
representation we can generate irreducible polynomial and is given in equation (6.3).
Irreducible Polynomial=x2+x+1
………………….6.3
Now the obtained polynomial x(1+x)= x2+x
is divided by irreducible polynomial of
x +x+1, then its remainder will be ‘1’ (i.e., 01). So the 01 is written in the multiplicative
2
table 6.3(c).
6.2.2 Representation of Polynomials

A polynomial of degree (n-1) is an expression of the form and is given in equation
(6.4)
f(x)= an-1xn-1+ an-2xn-2+ an-3xn-3+……… a1x1+ a0x0
………………….6.4
Where xi is called the ith term and ai is called coefficient of the ith term.
Example 6.4: Represent an 8 bit word by a polynomial for words 1 0 0 1 1 0 0 1.
Solution: Consider an 8 bit word to represent it in a polynomial. Hence we have

271
n-bit word
1 0 0 1 1 0 0 1
1x7 0x6 0 x5 1 x4 1 x3 0 x2 0 x1 1 x0 Polynomial
First simplification we have
1x7 1x4 1 x3 1 x0
Finally after second simplification, for 8 bit word, the generated polynomial is
x7 x4 x3 1
Example 6.5: For an 8 bit word represented a polynomial x5+x2+x.Find the bit words.
Solution: Since n=8. It means that the polynomial is of degree (n-1)-(8-1)=7. The
expanded polynomial is
0x7 0x6 1 x5 0 x4 0 x3 1 x2 1 x1 0 x0 Polynomial
The related bits words for 8-bit word is
0 0 1 0 0 1 1 0
6.2.3 Irreducible Polynomials:

For GF(2n) fields, the polynomials representing n-bit words use two fields such as
GF(2) and GF(2n). Thus it is noticed that 2n is not a prime number. So the concepts of
prime polynomials or irreducible polynomials are discussed in this section. For the
sets of polynomials in GF(2n), a group of polynomials of degree ‘n’ is defined as the
modulus. Such polynomials are referred to as irreducible polynomials. For the
different degree ‘n’ of polynomials, the irreducible polynomials are tabulated in
table 6.1.
272
Table 6.1: Irreducible Polynomials for degree n=1, 2, 3, 4, 5.

Degree (n) Irreducible Polynomials
1 (x+1), (x)
2 (x2+x+1)
3 (x3+x2+1), (x3+x+1)
4 (x +x +x2+x+1), (x4+x3+1), (x4+x+1)
4 3
5 (x5+x2+1), (x5+x3+x2+x+1), (x5+x4+x3+x+1), (x5+x4+x3+x2+1), (x5+x4+x2+x+1)
Example 6.6: Show that x2+1 is irreducible over the integer mod 7.
Solution: As we know that F=Z7={0, 1, 2, 3, 4, 5, 6} We can write
x2+1=(x+a) (x+b)
x2+1=(x2+xb+ax+ab)
where ‘a’ and ‘b’ belongs to F. Therefore,
x2+1= x2 +(a+b)x+ab
Comparing the coefficients of ‘x’ and constant ‘1’, we have
a+b=0
………………….6.5
ab=1
………………….6.6
If both the equations (6.5) and (6.6) are satisfied, then it is said to be reducible
polynomial otherwise, if any one of the equations (6.5) or (6.6) is not satisfied, then the
above polynomial is known as irreducible. So let us verify the polynomial for the
273
different combinations of mod 7. i.e., Z7={0, 1, 2, 3, 4, 5, 6} is (0, 0), (1, 6), (2, 5), (3, 4),
(4, 3), (5, 2) and (6, 1).
Equation 6.5 is satisfied when (a, b)= (0, 0)=0
(1, 6)=0 (1+6=7.i.e., 7 mod 7 =1 with remainder is 0)
Equation 6.6 is not satisfied when (a, b)= (0, 0)=0
(1, 6)=6 (1x6=6. i.e., since 6 is small to 7, it is not possible to divide)
(2, 5)=3 (2x5=10. i.e., 10 mod 7 =1 with remainder is 3)
(3, 4)=5 (3x4=12.i.e., 12 mod 7 =1 with remainder is 5)
(6, 1)=6 (6x1=6.i.e., since 6 is small to 7, it is not possible to divide)
Now it is recorded that only equation 6.5 is satisfied whereas equation (6.6) is not
satisfied. Hence it is proved that polynomial x2+1 is irreducible over the integer
mod 7.
Example 6.7: Find out the polynomial x3-9 is irreducible or reducible over the
integer mod 11.
Solution: As we know that F=Z11={0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10} We can write
x3-9=(x+a) (x2+bx+c)
274
where ‘a’, ‘b’ and ‘c’ belongs to F. Therefore,
x3-9= x3+bx2+cx+ax2+abx+ac
x3-9= x3+(a+b)x2+(ab+c)x+ac
Comparing the coefficients of ‘x2’, ‘x’ and constant, we have
a+b=0
………………….6.7
ab+c=0
………………….6.8
ac=2
………………….6.9
So x3-9=0 for x=4. Therefore x-4 or x+7 is a factor. We can see by long division
x3-9=(x+7)(x2+4x+5)
If all the equations (6.7), (6.8) and (6.9) are satisfied, then it is said to be reducible
polynomial otherwise, if any one of the equations (6.7) or (6.8) or (6.9) is not satisfied,
then the above polynomial is known as irreducible. So let us verify the polynomial for
the values of a=7, b=4 and c=5. Equation 6.7 is satisfied, when (a, b, c) = (7, 4, 5)=0
(7+4=11. i.e., 11 mod 11=1 with remainder is 0). Equation 6.8 is satisfied, when (a, b, c)
= (7, 4, 5)=0 (7x4+5=33. i.e., 33 mod 11=3 with remainder is 0). Equation 6.9 is not
satisfied when (a, b, c) =(7, 4, 5)=2 (7x5=35. i.e., 35 mod 11=3 with remainder is 2).
Now it is recorded that all equations (6.7), (6.8) and (6.9) are satisfied. Hence it is
proved that polynomial x3-9 is reducible over the integer mod 11.
275
6.3 Bose- Chaudhuri-Hocquenqhem [BCH] Codes:

The BCH codes are one of the most important and powerful classes of linear block
codes. These are the subset of cyclic codes whose generator polynomials have routes
carefully specified, so as to give good error correcting capability. Each BCH code is a
t-error correcting code where it can detect and correct up to‘t’ random errors per code
word. There are three main properties of BCH codes.
Table 6.2: Properties of BCH codes
(i) Block length of BCH codes is given in equation (6.10).
n=2m-1
………………….6.10
Where ‘m’ is any positive integer and its value is m
(ii) The number of message bits in BCH codes is given in equation (6.11).
k
………………….6.11
Where ‘n’ is block length, ‘m’ is any positive integer, ‘t’ is number of errors that the
BCH codes can detect and correct the errors.
(iii) The minimum distance for BCH codes (dmin) is given in equation (6.12).
dmin 2t+1
………………….6.12
6.3.1 Generator Polynomial:

For BCH codes, the generator polynomial g(x) is given by
g(x)=LCM[m1(x), m1(x),…….. m2t(x)]

………………….6.13
276
where ‘t’ is number of errors that BCH codes can detect and correct errors. For double
error correcting code, t=2 then the generator polynomial g(x) is given in equation
(6.14).
g(x)=LCM[m1(x), m2(x), m3(x), m4(x)]
………………….6.14
6.3.2 Degree of BCH Code (r):
The degree of BCH Code ‘r’ of (n, k) =n-k
Using equation (6.10) we have

K=2m-1-r
………………….6.15
Therefore the degree of code ‘r’ is given in equation (6.16)
K=n-r
………………….6.16
Example 6.8: Construct a triple error correcting BCH code with block length n=31
over GF(25).
Solution: It is given that ‘t’=3. So the generator polynomial g(x) is given by
g(x)=LCM[m1(x). m2(x). m3(x). m4(x). m5(x). m6(x)]

………………….6.17
In Galois field GF(2 ) we have,
5
m1(x)=x5+x2+1
m2(x)= m1(x)
m3(x)=x5+x4+x3+x2+1
m4(x)= m2(x)
m5(x)=x5+x4+x2+x+1
m6(x)= m3(x)
Now the generator polynomial g(x) after substitution of GF(25) values, we get
277
g(x)=LCM[m1(x). m3(x).m5(x)]
………………….6.18
Now to find the LCM of equation (6.18) we have
g(x)=LCM[(x5+x2+1). (x5+x4+x3+x2+1). (x5+x4+x2+x+1)]
Therefore after simplification we get
g(x)= x15+ x11+ x10+ x9+ x8+ x7+ x5+ x3+ x2+x+1
………………….6.19
From equation (6.10), we have
n=2m-1
31=2m-1
2m=32
Therefore
m=5
From equation (6.11), Information length
k=n-mt
After substitution we have
k=31-5x3
k=16
Therefore, we can say that it is (31, 16) triple error correcting BCH codes.
278
6.4 Reed-Solomon Codes (RS):

Reed- Solomon codes are block based error correcting codes with a wide range of
applications in digital communications and storage. RS codes are used to correct
errors in many systems such as storage devices, wireless or mobile communications,
satellite communications, digital television and high speed modems etc. Fig 6.4 shows
the typical system with RS encoder and decoder.
Fig. 6.4: Block diagram of an Typical System
The RS encoder takes a block of digital data and adds extra redundant bits. During
the communication channel, there may be errors are generated due to the various
reasons such as noise or interference, scratches on a CD etc.,. The RS decoder
processes each block and attempts to correct the errors and recover the original data.
The number and type of errors that can be corrected depends on the characteristics of
RS code.
6.4.1 Properties of Reed-Solomon Codes:
The important properties of RS Codes are tabulated in table 6.3.
279
Table 6.3: Properties of RS codes
(i) Reed Solomon codes are a subset of BCH codes and are linear block codes and it is
denoted as RS (n, k) with s-bit symbols.
(ii) The RS encoder takes ‘k’ data symbols of ‘s’ bits each and adds parity symbols to
make an ‘n’ symbol codeword.
(iii) There are (n-k) parity symbols of ‘s’ bits each.
(iv) A RS decoder can correct up to ‘t’ symbols that contain errors in a codeword.
Where 2t=n-k.
(v) A typical diagram of RS codeword is shown in Fig. 6.5
Data Parity
n
Fig.6.5: Typical diagram of RS codeword
It is also known as Systematic code because the data is left unchanged and the parity
symbols are appended.
Example 6.9: Describe a popular Reed-Solomon code is RS(255, 223) with 8-bit
symbols.
Solution: Each codeword contains 255 codeword bytes, of which 223 bytes are data
and 32 bytes are parity. For this code n=255, k=223, s=8, 2t=32, and t=16. This
indicates that the decoder can correct any 16 symbol errors in the codeword i.e., errors
up to 16 bytes positions in the codeword can be automatically corrected. It is given
that symbol size ‘s’ the maximum codeword length (n) for a Reed-Solomon code is
280
n=2s-1. This means that the maximum length of a code with 8-bit symbols (s=8) is 255
bytes. Reed Solomon codes may be shortened by making a number of data symbols
zero at the encoder (not transmitting them) and then re-inserting them at the decoder.
Example 6.10: For the (31, 15) Reed Solomon code.

(i) Find how many bits are presented in a symbol.
(ii) Find the block length in bits.
(iii) Obtain the minimum distance of the code.
(iv) How many symbols in error can the code correct?
(v) Find the length of an in phase that code can correct?
Solution: For Reed-Solomon codes

(i) The number of bits per symbol in the code is given by
n=2m-1
Since n=31. Therefore
31=2m-1
32=2m
25 =2m
m=5
(ii) Block length is obtained by
nxm
31 x 5
155 bits
281
(iii) The minimum distance of the code is given by
dmin=2t+1
Since we have 2t=n-k. Therefore
dmin=n-k+1
dmin=31-15+1
dmin=17
(iv) The number of correctable symbols which are in error
t=
t=
Therefore
t=8
(v) The code can correct an in phase burst of length
m=5
6.4.2 Generator Polynomial:

A Reed-Solomon codeword is generated using a special polynomial. All valid code
words are exactly divisible by the generator polynomial. The general form of
generator polynomial is given in equation (6.20).
g(x)=(x-ai)(x-ai+1)……… (x-ai+2t)
………………….6.20
282
And the code word is constructed using
c(x)=g(x) i(x)
………………….6.21
Where g(x) is the generator polynomial, i(x) is the information block, c(x) is a valid
codeword and is referred to as a primitive element of the field.
6.5 Golay Codes:

A Golay code is a perfect linear error correcting code. There are two essentially
distinct versions of Golay code such as binary and ternary versions. These codes are
named in honor of Marcel. J. E. Golay. There are two closely related binary Golay
codes. The extended binary Golay code, G24 encodes 12 bits of data in a 24 bit word in
such a way that any three bit errors can be corrected or any seven bit errors can be
detected. The standard notation for extended binary Golay code parameters [24, 12, 8]
where 24 indicates length of the code words, 12 indicates the dimension of the code
and 8 represents the minimum hamming distance between two code words. The
perfect binary Golay code, G23 has code words of length 23 and is obtained from the
extended binary Golay code by deleting one coordinate position. The standard
notation for extended binary Golay code parameters [23, 12, 7] where 23 indicates
length of the code words, 12 indicates the dimension of the code and 7 represents the
minimum hamming distance between two code words.
Example 6.11: Show that the (23, 12) binary code is a perfect Golay code.
Solution: For the (23, 12) Golay code that has capable of correcting up to three errors
in a block of 23 bits. It is a perfect binary code, since it satisfies the Hamming bound
with equality sign for t=3. For Golay code consider
2k=∑ = 212 ∑
283
= 212 ∑
= 212 (2048)
=212. 211
=223
Therefore
2k=∑ =2n
284
THINGS TO BE
REMEMBER
1. Galois Field (GF) demonstrates 6. Each BCH code is a t-error

that, for the field to be finite the correcting code where it can detect
number of elements should be pn, and correct upto ‘t’ random errors
where ‘p’ is a prime and ‘n’ is a per code word.
positive integer.
7. Block length of BCH codes is
2. For GF(2n), the ‘n’ can hold the given by
values of 1, 2, 3…..which has a n=2m-1
particular value that plays a 8. The minimum distance for BCH
prominent role in network security. codes (dmin) is given by
dmin 2t+1
3. A polynomial of degree (n-1) is an
expression of the form and is given 9. For BCH codes, the generator
by polynomial g(x) is given by
f(x)=an-1xn-1+ an-2xn-2+ an-3xn-3+… g(x)=LCM[m1(x), m1(x),…….. m2t(x)]

a1x1+ a0x0
10. A Golay code is a perfect linear
4. For the sets of polynomials in error correcting code.
GF(2n), a group of polynomials of
degree ‘n’ is defined as the
modulus. Such polynomials are
referred to as irreducible
polynomials.
5. The BCH codes are one of the most

important and powerful classes of
linear block codes.
285
References
[1] R. Johannesson and K. S. Zigangirov, Fundamentals of Convolutional Coding, 2nd
Edition, Wiley‐IEEE Press, 2015.
[2] E. Biglieri, D. Divsalar, P. J. McLane and M. K. Simon, Introduction to Trellis Coded
Modulation with Applications, Macmillan, 1991.
New York; 1989.
Delhi; 2008.
and Sons: 1989.
286
Chapter 7
Convolution Codes
Syllabus: Convolution Codes: Convolution Encoder, Time domain approach, Transform
domain approach, Code Tree, Trellis and State Diagram, The Viterbi Algorithm.
Contents
7. Convolution Codes Page. No
Learning Objectives
7.1 Introduction...........................................................................................................................281
7.2 Some definitions............................................................................................................. .......282
7.2.1 Code rate.............................................................................................................. .........282
7.2.2 Constraint length.........................................................................................................282
7.3 Analysis of convolution encoder using time domain approach......................................282
7.4 Analysis of convolution encoder using matrix method...................................................288
7.5 Analysis of convolution encoder using transform domain approach............................290
7.6 Graphical Approaches...........................................................................................................301
7.6.1 State diagram................................................................................................................301
7.6.2 Code tree.......................................................................................................................305
7.6.3 Trellis diagram.............................................................................................................307
7.7 Viterbi algorithm for convolution decoding......................................................................309
7.7.1 Some definitions...................................................................................................... ....310
References
287
I visualize a time when we will
be to robots what dogs are to
humans, and I am rooting for
the machines.
Claude. E. Shannon
288
7 CONVOLUTION CODES
Learning Objectives

Introduce binary convolution codes in terms of shift register encoders.
Formalize description using formal power series and generator matrix.
Discuss the trellis representation and Viterbi decoding algorithm.
Engage the students in active learning through problem solving and real world examples.
Encourage the student to become an independent learner and increase his/her awareness of
available resources.
7.1 Introduction
In the previous section we have discussed error control codes which are basically
classified into two categories as block codes and the convolution codes. Block codes
have k-message bits which are encoded into an n-bit code vector whereas convolution
codes generate parity symbols and works on input bits rather than input sequence. In
convolution codes, coding is done by combining the input bits which are in fixed
number.
Peter Elias in the year 1995 introduced convolution codes that are used to achieve
in various applications to transfer the data more effectively such as in digital videos,
radios, mobile and satellite communications. The difference between block codes and
convolution codes is entire input data sequence is encoded into code words in
convolution codes whereas in case of block codes, input data sequence gets
segmented into blocks and individual block gets encoded. Other difference is unlike
block codes convolution codes possess memory. The convolution code generates „n‟
symbol code words from „k‟-symbol message blocks. The Convolution encoder has m-
stage shift registers acts as a memory elements and a circuit to perform operation to
generate the code words. It says that the output is dependent on previous inputs as
well. Hence, the convolution codes are implemented not only by „n‟ and „k‟ values but
also the „m‟ memory elements used in encoder. So, the implementation of the
289
convolution codes can be written as (n, k, m) where „n‟ indicates the number of
outputs, „k‟ indicates inputs and the „m‟ indicates memory elements.
7.2 Some Definitions

In convolution encoder to determine the number of outputs length for the input bits
let us consider some important definitions namely code rate and constraint length.
7.2.1 Code Rate (r)

Code rate is used to determine the number of data bits per code bit i.e., number of
inputs per number of outputs. For (n, k, m) convolution encoder the code rate is given
in equation (7.1).
r=k/n
………………….7.1
Where „k‟ indicates number of inputs and „n‟ indicates number of outputs.
7.2.2 Constraint Length (K)

Constraint length is defined as the number of successive shifts of single message
which influences the encoded output. It is denoted by „K‟.
7.3 Analysis of Convolution Encoder Using Time Domain Approach

For LTI systems the output is obtained by convolving the input sequence with the
impulse response of the system. Let us assume that a message sequence m= {m0, m1,
m2,...........,mL-1} of „L‟ length and code rate is ½ and the constraint length K=3.
Fig. 7.1: Convolution Encoder (2, 1, 2)

290
Convolution codes
Let us consider a sequence {g0(1), g1(1), g2(1),........, gm(1)} of impulse response of the
module-2 adder which generates an output x1 and similarly for an output x2 the
impulse response or generator sequence be {g0(2), g1(2), g2(2),........, gm(2)}. The resulted
output will be the combined impulse response of both the output1 and output2 and is
called as the generator sequence of the code. By considering the Fig 7.1, we have two
outputs i.e., n=2 and two memory elements m=2 with message bits k=1. Here, we
need to find the number output bits for both module-2 adders. Hence, for both the
outputs x1 and x2 we have to find the generator sequence.
To find the generator sequence

Generator sequences are impulse responses of the system. Thus impulse response is
the output sequence of an adder for the given input sequences. Here we have to find
out the impulse responses of the „n‟ outputs i.e., 2 outputs. From the Fig 7.1, the
generator sequence is given by the connections to the output 1 (i.e., the input and the
outputs of the first memory element) and second memory elements are directly
connected. So the generator sequence g(1) and g(2) are given as,
g(1) =( g0(1), g1(1), g2(1))=(1, 1, 1)
Similarly, for the output 2 generator sequence is
g(2) =( g0(2), g1(2), g2(2))=(1, 0, 1)
To find the output sequence

Output sequence can be calculated by doing convolution of the message sequence
with the corresponding generator sequences. Let us consider the output sequence
for x from the discrete convolution method as
(1)
x(i)(1)=∑
.............................7.2
Where
i= 0, 1, 2........indicates the bit index.
M = length of the generator sequences.
= message length.
Similarly for the output sequence for x(2)
291
x(i)(2)=∑
.............................7.3
At the output side there is a multiplexer which switches between the two outputs. So,
the final output {Xi} is the multiplexed output of the two outputs i.e., one bit from x(i)(1)
and next bit from x(i)(2).
{Xi} ={ x(0)(1) , x(0)(2) , x(1)(1) , x(1)(2) ,..........}
.............................7.4
The length of the output should be L+M-1. Where, L and M indicates the length of the
message sequence and length of the two generator sequences respectively. To
understand the time domain approach let us consider the example 7.1.
Example 7.1: For the convolution encoder given in Fig 7.1, find the generator
sequence and the output sequence for the given message sequence m = {1 1 1 0 1} with
a convolution rate 1/2.
Solution: From the given Fig 7.1 it is noticed that there are two output bits i.e., n=2,
memory elements m=2 and the input bit k=1. So we can say that the convolution
encoder is of (2, 1, 2) code. So the generator sequence g(1) and g(2) are given as
g(1) =( g0(1), g1(1), g2(1))=[1, 1, 1]

Similarly,
g(2) =( g0(2), g1(2), g2(2))=[1, 0, 1]
For the given message sequence m= {m0, m1, m2, m3, m4} of length L= 5 and the length
of the two generator sequence m=3. So, the output sequence length is
L+M-1=5+3-1=7bits.
From the equation (7.2.1), the output sequence bit for x(1) is given by
x(i)(1)=∑
292
Convolution codes
Now, for i=0,

x(0)(1)=∑ = +
Since we are not having the message bit value for so the first bit of the first adder
is only consist of
x(0)(1)=
Now by substituting the values of . We get
x(0)(1)= = (1 x 1)=1
For i=1,
x(1)(1)=∑ = +
= (1 x 1) + (1 x 1) = 1 + 1 =0
For i=2,
x(2)(1)=∑ = + +
= (1 x 1) + (1 x 1) + (1 x 1) = 1 + 1 + 1=1
For i=3,
x(3)(1)=∑ = + +
= (1 x 0) + (1 x 1) + (1 x 1) = 0 + 1 + 1=0
For i=4,
x(4)(1)=∑ = + +
= (1 x 1) + (1 x 0) + (1 x 1) = 1 + 0 + 1=0
For i=5,
x(5)(1)=∑ = + +
Since, the values of m5 is not there we will consider x(5)(1) as
x(5)(1)= +
293
= (1 x 1) + (1 x 0)= 0 + 1 + 0=1
For i=6,
x(6)(1)=∑ = + +
Since, the values of m5 and m6 are not there we will consider x(6)(1) as
x(6)(1)=
= (1 x 1) = 1
NOTE: The value of i= n x m where n indicates the output bits and m indicates the
bits of generator sequence.
Similarly, From the equation (7.3), the output sequence bit for x(2) is given by
x(i)(2) =∑
Now, for i=0,
x(0)(2)=∑ = +
Since we are not having the message bit value for so the first bit of the first adder
is only consist of x(0)(2)= . Now, by substituting the values of . We
get
x(0)(2)= = (1 x 1)=1
For i=1,
x(1)(2)=∑ = +
= (1 x 1) + (0 x 1)= 1 + 0 =1
For i=2,
x(2)(2)=∑ = + +
= (1 x 1) + (0 x 1) + (1 x 1)= 1 + 0 + 1=0
For i=3,
294
Convolution codes
x(3)(2)=∑ = + +
= (1 x 0) + (0 x 1) + (1 x 1)= 0 + 0 + 1=1
For i=4,
x(4)(2)=∑ = + +
= (1 x 1) + (0 x 0) + (1 x 1)= 1 + 0 + 1=0
For i=5,
x(5)(2)=∑ = + +
Since, the values of m5 is not there we will consider x(5)(1) as
x(5)(2)= +
= (0 x 1) + (1 x 0)= 0 + 0 =0
For i=6,
x(6)(2)=∑ = + +
Since, the values of m5 and m6 are not there we will consider x(6)(1) as
x(6)(2)=
= (1 x 1)=1
Finally, the two output bits are written as
x(1) ={1 0 1 0 0 1 1}
x(1) ={1 1 0 1 0 0 1}
The output sequence can be written from the equation (6.3) as
{Xi} ={ x(0)(1) , x(0)(2) , x(1)(1) , x(1)(2) ,..........}
{Xi}= {11 01 10 01 00 10 11}
The message sequence is of L=5 and the code rate is 1/2 which states that for 1 input
bit it should produce 2 output bits. But the output bits are of length 14 .For each
295
message two output bits, for the last four bits the input is considered as 0 and 0. This
resets the encoder for the further operations.
7.4 Analysis of Convolution Encoder Using Matrix Method

The matrix method is used to calculate the output of the convolution codes if the
length of input message sequence is short. The following steps are used to create a
matrix of the generator sequences. Let us consider a two generating sequences
and .
={ ...... }
={ ...... }
Now let the message sequence m= {m1, m2, m3,....., mL} of length „L‟. The generator
matrix has „L‟ rows and n(L+m-1) columns. Hence, the matrix formed as
G=
[ ]
.............................7.5
Hence, the output of the convolution encoder „X‟ is written as
X= m x G
.............................7.6
X=[ m1, m2, m3,....., mL] x G
X=
[ ]
By performing the multiplication, final output of the convolution encoder is obtained.
296
Convolution codes
Example 7.2: For the convolution encoder (2, 1, 2), the impulse response is
=[1 1 1] and =[1 0 1]. Construct the generator matrix and also find the
output code vector for the given message sequence m= [1 0 0 1 1].
Solution: For (2, 1, 2) convolution encoder the output bits n=2, memory elements
m=2, the input bit k=1 and the impulse response sequence bit M=3. The generator
sequences are
={ } =[ 1 1 1]
={ } =[ 1 0 1]
The message sequence
m= {m1, m2, m3, m4, m5} = [1 0 0 1 1] with length L=5.
Generator Matrix will be having „L‟ rows i.e., 5 rows and the n x (L+M-1) columns i.e.,
2 x (5+3-1) = 14columns. Therefore, generator matrix „G‟ can be written as
G=
[ ]
Now, by substituting the values in the matrix „G‟ will become
G=
[ ]
From the equation (7.6) the output sequence „X‟ is given by
X=m x G
Hence, the output sequence will become
297
X= [1 0 0 1 1]
[ ]
X= [11 10 11 11 01 01 11]
Note: Two rows of the generator matrix is the output for the one message bit.
7.5 Analysis of Convolution Encoder Using Transform Domain Approach

In Transform Domain approach of the convolution encoder is used to express the
message sequence and the generator sequence in terms of polynomials as a function
of delay parameters. Expressed polynomials undergoes for the module-2
multiplication in order to get the desired output sequences. Steps in transform
domain approach are as follows:
Step 1: Express the generator sequences as polynomials as a function of unit delay
parameter „D‟. In general the generator sequence for the first output is given by
={ }
Now, express the generator parameter in terms of polynomial as a function of unit

delay parameter „D‟.
=
………………7.7
If we look at the equation (7.7) we can notice that the coefficients of the polynomial
are the generator sequences. Similarly, for the second output generator sequence is
given by
={ }
By expressing the generator parameter in terms of polynomial as a function of unit

delay parameter „D‟ we get
298
Convolution codes
=
………………7.8
Step 2: Express the message sequences as polynomials. In general the message
sequence can be written as
m= {m0, m1, m2,……, mL-1} of a length „L‟ bits
Now, express the message sequence as a polynomial of the function „D‟.
m(D)=
………………7.9
Step 3: Calculate the output as a polynomial. Let the outputs be (D) and (D)
when expressed as a function of delay parameter „D‟. So, we can write the first output
of the convolution encoder as the product of the message sequence and the first
generator sequence with the delay parameter „D‟.
(D)= m (D)
………………7.10
Similarly, the second output of the convolution encoder is the product of the message
sequence and the second generator sequence with the delay parameter „D‟.
(D)= m (D)
………………7.11
{Xi} ={ x(0)(1) , x(0)(2) , x(1)(1) , x(1)(2) ,..........}
........................7.12
The sequences of the output can be written by taking the one bit from the first output
sequence and the first bit from the second output sequence. Hence, the equation
(7.10) and (7.11) gives the final encoded output equation (7.12) of the convolution
code in transform domain method.
299
Example 7.3: For the convolution encoder (2, 1, 2), the impulse response is
= [1 1 1] and =[ 1 0 1]. Find the output code vector for the given message
sequence m= [1 0 0 1 1] with a code rate ½ and the Constraint length K=3.
Solution: From the given data for (2, 1, 2) convolution encoder the output bits n=2,
memory elements m=2, the input bit k=1.To obtain the output code vector the
following steps are followed.
Step 1: Express the generator sequences as polynomials as a function of unit delay

parameter „D‟. The given generator sequence for the first output is
={ = [1 1 1]
Now express the generator parameter in terms of polynomial as a function of unit

delay parameter „D‟.
=
By substituting the values of impulse response in the above equation we get
= =
This is the first generator sequence polynomial. Similarly, for the second output
generator sequence is given by
={ }=[1 0 1]
By expressing the generator parameter in terms of polynomial as a function of unit

delay parameter „D‟ we get
=
By substituting the values of impulse response in the above equation we get
= =
300
Convolution codes
This is the second generator sequence polynomial.
Step 2: Express the message sequences as polynomials. The given message sequence is
m= {m0, m1, m2, m3, m4} =[1 0 0 1 1] of a length L=5 bits
m (D)=
By substituting the values of the message sequence in the above equation we get
m (D)=
m (D)= 1+ +
This is the message sequence polynomial.
Step 3: Calculate the output as a polynomial. From the equation (7.10) we can write
the first output of the convolution encoder as the product of the message sequence
polynomial and the first generator sequence polynomial with the delay parameter „D‟.
(D)= m (D)
Now, by substitute the values of m (D) and we get
(D)= (1+ + )( )
= 1+D+ + + + + + +
=1+ D+ + + (1+1)+ (1+1)+
(D)= 1+ D+ + +
301
Express the polynomial in terms of output sequence by looking at the coefficients of

the polynomial.
=[1 1 1 1 0 0 1]
Similarly from the equation (7.11) the second output of the convolution encoder is the
product of the message sequence and the second generator sequence with the delay
parameter „D‟.
(D)= m (D)
Now, by substitute the values of m (D) and we get
(D)= (1+ + )( )
= 1+ + + + +
(D)= 1+ + + +
Express the polynomial in terms of output sequence by looking at the coefficients of

the polynomial.
= [ 1 0 1 1 1 1 1]
The final encoded output sequence is written as
X= [11 10 11 11 01 01 11]
Example 7.4: For the convolution encoder (3, 1, 2) with the impulse response is
=[1 1 0], =[1 0 1] =[1 1 1].
302
Convolution codes
(i) Draw the block diagram of an encoder.

(ii) Determine the code rate.
(iii) Construct a generator matrix and find the code vector for m= {1 1 1 0 1}.
(iv) Analyze the output vector using both the time domain method and
transform domain method.
Solution: (i) Convolution encoder can be constructed from the given impulse
response =[1 1 0], =[1 0 1] =[1 1 1]. The (3, 1, 2) convolution encoder
has output bits i.e., number of module -2 adders (n=2), number of memory elements
m=3 and the input bits (k=1).
Fig. 7.2: Convolution Encoder (3, 1, 2)
(ii) From the equation (7.1) the code rate is defined as the ratio of input bits to the
output bits i.e.,
R= k/n= 1/3
It says that for every single input the encoder will produce output of three bits.
(iii) To construct a generator matrix we have impulse responses. We have impulse
responses = [ 1 1 0], =[ 1 0 1] =[ 1 1 1] .The number of rows is equal
303
to number of message bits i.e.,L=5 and number of columns is n x (M+L-1)= 3 x (3+5-

1)=21.
G=
[ ]
The output of the encoder for the message sequence m=[11101] is given by
X= m x G
X=[11101]
[ ]
X=[111 010 001 110 100 101 011]
(iv) Analyze the output using time domain approach is as follows
For adder 1 i.e., from the equation (7.2) the output X is given by
x(i)(1)=∑
For i=0
x(0)(1)=∑ g
x(0)(1)=g
x(0)(1)=1x1=1
Similarly, for i=1, 2, 3, 4, 5, 6. We have
x(1)(1)=g +g =(1 x 1)+(1 x 1) = 0
304
Convolution codes
x(2)(1)=g +g g = (1 x 1)+(1 x 1)+(0 x 1) = 0
x(3)(1)=g +g g = (1 x 0)+(1 x 1)+(0 x 1) = 1
x(4)(1)=g +g g = (1 x 1)+(1 x 0)+(0 x 1) = 1.
x(5)(1)=g g = (1 x 1)+(0 x 0) = 1
x(6)(1)=g = (0 x 1) = 0
For adder 2 i.e., from the equation (7.3) the output X is given by
x(i)(2)=∑
For i=0
x(0)(2)=∑
x(0)(2)=g
x(0)(2)=1x1=1
x(1)(2)=g +g =(1 x 1)+(0 x 1) = 1
x(2)(2)=g +g g = (1 x 1)+(0 x 1)+(1 x 1) = 0
x(3)(2)=g +g g = (1 x 0)+(0 x 1)+(1 x 1) = 1

x(4)(2)=g +g g = (1 x 1)+(0 x 0)+(1 x 1) = 0
x(5)(2)=g = (0 x 1)+(1 x 0) = 0
305
= = (1 x 1) = 1
For adder 3,
=∑
For i=0
=∑
=1x1=1
= + =(1 x 1)+(1 x 1) = 0
= + = (1 x 1)+(1 x 1)+(1 x 1) = 1
= + = (1 x 0)+(1 x 1)+(1 x 1) = 0
= + = (1 x 1)+(1 x 0)+(1 x 1) = 0
= = (1 x 1)+(1 x 0) = 1
= = (1 x 1) = 1
The final output is calculated by multiplexing , and
X= [111 010 001 110 100 101 011]

Analyze the output using time domain approach is as follows:
= [ 1 1 0], =[ 1 0 1] =[ 1 1 1].
306
Convolution codes
Express the generator sequences as polynomials as a function of unit delay parameter

„D‟.
={ =[1 1 0]
=
,
= =
={ =[1 0 1]
= =
={ =[1 1 1]
= =
Express the message sequences as polynomials. The given message sequence is
m= {m0, m1, m2, m3, m4} =[1 1 1 0 1] of a length L=5bits
m (D)=
By substituting the values of the message sequence in the above equation we get
m (D)=
m (D)= 1
307
To obtain the output at adder 1

X(1)D= m (D)
=( )( 1 )
=1+D+ + + + +
=1+D(1+1)+ (1+1)+ + +
X(1)D=1+ + +
The sequence is
X(1)=[1 0 0 1 1 1 0]
X(2)D= m (D)
=( )+( 1 )
=1 + + + +
=1+D+ (1+1)+ + (1+1)+
X(2)D=1+D+ +
The sequence is
X(2)=[1 1 0 1 0 0 1]

X(3)D= m (D)
=( )( 1 )
=1 + + + + + + + +
=1+D(1+1)+ (1+1)+ (1+1)+ (1+1)+ +
308
Convolution codes
X(3)D=1+ +
The sequence is
X(1)=[1 0 1 0 0 1 1]
The multiplexed output will be
X= [111 010 001 110 100 101 011]
7.6 Graphical Approaches:

In the previous sections the output of the convolution encoder was analyzed using
time-domain approach and the transform–domain approach. This section deals with
the state diagram and the code tree to analyze the encoder results. Convolution
encoding can be done using one of the graphical approaches.
(i) State Diagram
(ii) Code tree
(iii) Trellis Diagram
7.6.1 State diagram
State diagram indicates the number of transitions flow in the network. To draw the
state diagram the following steps need to be followed:
Step 1: Determine the number of transition states.
Number of states = 2m where m= number of memory elements
If the convolution encoder is of (2, 1, 2) then the memory elements m=2. Then the
states are 22= 4 states.
Step 2: write down the states with respect to the combination of memory elements i.e.,
for convolution code (2, 1, 2).
Table 7.1 State Table
m1 m2 State
0 0 S0
309
0 1 S1
1 0 S2
1 1 S3
Step 3: Construct the state transition table by considering the states and the possible
input bits of the states. Also determine the output of the encoder by considering the
inputs and memory elements.
Step 4: Draw the state diagram with respect to the state transition table.
Example 7.5: Consider the convolution encoder (2, 1, 2) as shown in Fig. (7.3) below
and the impulse response is g(1)=[111]and g(2)=[101] Draw the state diagram.
Fig. 7.3: Convolution Encoder

Solution: To construct the state diagram the following steps are as follows:
Step 1: find the transition state. Since from the convolution code (2, 1, 2) is having
memory elements m=2.
Number of transition states =2m =22= 4 states
310
Convolution codes
Step 2: Write down the states with respect to the combination of memory elements
i.e., for convolution code (2, 1, 2).
Table 7.2: State Table

m1 m2 State
0 0 S0
0 1 S1
1 0 S2
1 1 S3
Step 3: Construct the state transition table. From the Fig (7.3) let us consider d1, d2
and d3 as input value and output of the memory element values X(1) and X(2) are given
as
X(1)= d1 + d2 + d3
X(2)= d1 + d3
Table 7.3: State transition Table

Input Present State Next State Output
State Value State Value X(1) X(2)
0 S0 00 S0 00 0 0
1 00 S1 10 1 1
0 S1 10 S2 01 1 0
1 10 S3 11 0 1
0 S2 01 S0 00 1 1
1 01 S1 10 0 0
0 S3 11 S2 01 0 1
1 11 S3 11 1 0
Calculating the output:
Let us consider input column values as d1 and values of the present states as d2 and
d3. Now, calculate the output for
X(1)= d1 + d2 + d3 = 0 + 0 + 0 =0
X(2)= d1 + d3 = 0 + 0 + 0 = 0
311
So, the output for the state S0 is “00”when the input bit „0‟. Similarly id the input bit is
„1‟ then the output will be
X(1)= d1 + d2 + d3 = 1 + 0 + 0 =1
X(2)= d1 + d3 = 1 + 0 + 0 = 1
The output is 11. Similarly calculate the output for all the remaining states.
Step 4: Draw the state diagram with respect to the state transition diagram.
From the table 7.3 of state transition table , it is clear that when the present state „S0‟ is
input is „0‟ then the state transit to the next state „S1‟ by producing output “00” and for
input ‟1‟ transit to the next state „S2‟ with output “11”. Similarly for the remaining
states we have
Fig. 7.4 : Convolution Encoder

7.6.2 Code tree
A tree diagram is redrawn model of the state diagram considering the branches of the
tree. It is the most effective and efficient way to represent the encoder rather than the
state diagram. The features of the code tree are:
(i) Every input bit either „0‟ or „1‟ represents a branch of the tree.
312
Convolution codes
(ii) The upper divided branch of the tree represents a bit „0‟.
(iii) The lower divided branch of the tree represents a bit „1‟.
(iv) The path to trace the output of the encoded sequence should start from left
side to the right side.
(v) After every three branches the tree structure will repeat for the further
bifurcation.
Construction of code tree:

Step 1: To draw the code tree let us consider the table 6.2 of state transition table for
the convolution code (2, 1, 2).
Step 2: Start with the initial state „S0‟. the state has two input conditions „0‟ and „1‟, if
the input bit is „0‟ move to the upward branch that state is „S0‟ with the corresponding
output value “00”.
Step 3: If the input value of the state „S0‟ is „1‟ then move to the downward of the
branch „S1‟ with the corresponding output value “11”.
Step 4: Divide the branches according to the states by referring to the table 6.2 of state
transition table with their output values for the corresponding input value.
Step 5: The bifurcation of the tree branches continues till the length of message input
stages i.e., „L‟ stages. The Fig (7.5) illustrates the tree diagram for convolution encoder
(2, 1, 2).
313
Fig. 7.5: code tree
314
Convolution codes
Tracing the Encoded output:

Let us consider the input sequence m={1 0 0 1 1}. To trace the output, the first input bit
is „1‟ trace should move to the downward branch from the initial. Note down the
corresponding output value “11”. Now, the second input bit is „0‟ the tracing should
move to the upward direction from the stopped place, note down the respective
output value “10”. Similarly, continue the tracing till the L message bits. Now the
encoded output will be [11, 10, 11, 11, 01]. From the previous outputs with the time
domain approach and transform domain approaches there we have 14 output bits. So,
to achieve the output we need to take the input bits as „0‟. The added bits are called as
the tail bits, in the diagram again the branch bifurcates with respect to the tail bits.
The final traced output will be [11 10 11 11 01 01 11].
7.6.3 Trellis Diagram

A tree with a merged branch is called trellis structure. Trellis is a compact form of a
code tree. The trellis posses a following features:
(i) There are L+K levels. Where L is the length of the message sequence and the K
is the constraint length of the encoder.
(ii) Starting (K-1) levels are used for departure of the encoder from the initial
state.
(iii) Last (K-1) levels are to return to the initial state with tail bits as „0‟.
(iv) The structure is periodic in the levels K-1 ≤ J ≤ L.
Construction:
Let us consider a convolution encoder (2, 1, 2) with the message sequence m = {1 0 0 1
1 } of L length and the constraint length K=3. The structure of trellis is as follows.
(i) There are L+K levels i.e., 5+3=8 levels. Index bit is denoted as J. So, the level of
indexing will be J=0, 1, 2, ….., L+K-1. i.e., from J=0 to J=7 levels.
(ii) The input bit „0‟ is indicated with solid line and the input „1‟ indicated with the
dashed line. The corresponding outputs are written in the lines.
0
315
Fig. 7.6: Trellis Diagram
Initially we are at the stage „S0‟ of level J=0, for the input bit „0‟ the next state will be
„S0‟ draw the solid line with corresponding output “00”. Similarly, for the input bit „1‟
the next state is „S1‟ draw the dashed line with the corresponding output “11”. The
process will continue from the level J=1. We can observe here that the initial level j=0
is departure in to two stages at the level j=1.
(iii) At the level K-1 i.e., J=2 we got all the states connected and completely
departure from the initial state.
(iv) Now from the Fig (7.6) the structure is periodic from the K-1 to L level i.e.,
from j=2 to j=5.
(v) We can also observe that at the last K-1 levels i.e., J=7, the structure is similar
like the initial structure.
316
Convolution codes
Tracing:
For the input sequence m= {1 0 0 1 1} , tracing will start from the level J=0 . As the first
bit is „1‟ output of the dashed line of that branch will be considered i.e., “11”. For the
next message sequence bit „0‟ the corresponding output of the solid line at the stage
J=1 will be considered. Similarly, the process will repeat for the remaining message
sequences. The level J=6 and J=7 indicates the tail bit „0‟ the output of tail bits is also
considered. The final encoded output is written as [ 1 1 1 0 1 1 11 01 01 11]. The
periodic structure of trellis can be given in Fig (7.7) shown below.
Fig. 7.7: Periodic Trellis diagram
7.7 Viterbi Algorithm for convolution decoding:

In the year 1967 the convolution decoding method Viterbi Algorithm was developed
by Andrew J. Viterbi. Viterbi Decoding is also called as the maximum likelihood
decoding method. Viterbi decoding method has some advantages which are listed
below as follows
(i) High performance rate in correcting the error in the received bit sequence.
(ii) Operation to perform the error correction is high.
(iii) Easy to implement.
(iv) Cost effective to implement.
(v) The shortest path in the network with the weights can be determined.
7.7.1 Definitions:
317
Branch Metric: The hamming distance between the particular branch output and the
frame length of the particular received bit. For example if the output of the first
branch is “00” and the first frame of the received sequence is “10”then the Branch
metric or hamming distance d(00, 10) is „1‟.
Path Metric: The total metric accumulated by the each branch at the node is called as
the path metric.
Surviving Node: It is the path of the decoding sequence with the minimum path
metric.
Let us consider the received sequence R= [11, 10, 11] of 6 bits, decode the encoder (2,
1, 2). Assume that the states S0 as a0, S1 as b0, S2 as c0 and S3 as d0. Now, the decoder is
at S0 state means a0 state, from the trellis periodic diagram as shown in Fig 7.7 the
branch will bifurcate in two a0 and b0. The division of the branch continues according
to the trellis diagram with their corresponding output bits. The Fig 7.8 shows the
Viterbi decoding structure.
Fig.7.8: Viterbi Decoding Diagram
318
Convolution codes
From the Fig. (7.8) we can observe that each

THINGS TO BE
level is written with the frame of received bits.
REMEMBER Now, calculate the hamming distance between
the received bit and the output of the corresponding level of the branch i.e., the initial
branch is bifurcated into a0 and b0 the hamming distance between the received bit and
the branch output d(11, 00) is 2. Similarly, calculate the branch metric for all the
branches. Calculate the path metric at the each node by adding the branch metric.
Table (7.4) gives the surviving path with corresponding distance.
Table 7.4 Path metric table
Path Path metric
a0 – a1 – a2 – a3 2+1+2=5
a0 – a1 – a2 – b3 2+1+0=3
a 0 – a 1 – b2 – c3 2+1+1=4
a 0 – a 1 – b2 – d3 2+1+1=4
a 0 – b1 – c 2 – a3 0+0+0=0
a 0 – b1 – c 2 – b3 0+0+2=2
a 0 – b1 – d 2 – c3 0+2+1=3
a 0 – b1 – d 2 – d3 0+2+1=3
From the Table (7.4) of path metric we can observe that the branch metric from a0 to
a1 is 2, a1 to a2 is 1 and a2 to a3 is 2 so the path metric will become 5 shown at the node
a3. So, we are going to neglect that path. Similarly, by calculating all the path metric
value we observe from the table that the path a0 – b1 – c2 – a3 is going to be the
surviving path with the minimum path metric value „0‟. To find the value of the
decoder trace the surviving path which possesses the least branch metric value. From
the Fig (7.8) by tracing the surviving path the branch metric value of a0 – b1 is 0, for the
input bit „1‟ indicated with the dashes line, branch metric value of b 1 – c2 is 0, for the
input bit „0‟ indicated with the solid line and the branch metric value of c 2 – a3 is „0‟,
for the input bit „0‟ indicated with the solid line. Hence, the decoded value is [1, 0, and
0].To verify the answer check the first three message sequence values for the first six
bits of the encoded output from the previous examples.
319
1. Convolution encoder is denoted by 7. Viterbi Decoding Algorithm is also

(n, k, m) where n=number of outputs, known as the Maximum
k= number of inputs and m=number of Likelihood Decoding.
memory elements.
2. Code rate ‘R’ is defined as the ratio

of number of input bits to the number
of output bits.
R= K/n
3. Constraint length is the maximum

number of output bits occurs by a
single input bit.
4. Analysis of the convolution

encoder output can be done by
Generator Matrix
Time domain approach
Transform Domain Approach
5. Graphical representation is done by

State Diagram
Tree diagram
Trellis Diagram
6. Decoding of the convolution code is

done by viterbi decoding method.
320
Convolution codes
DESCRIPTIVE
QUESTIONS
1. For the (3, 1, 2) convolution encoder (iii)Using the time domain approach
with the generator sequences g(1) =[1 find the output sequence.
1 0], g(2) =[1 0 1], and g(3) =[1 1 1] (iv)Using the transform domain
(i) Draw the block diagram of approach find the output sequence.
encoder.
(ii)For the message sequence m= {1 4. Construct the state diagram and
1 1 0 1} construct the generator trellis diagram for the convolution
matrix and the find the output code encoder (2, 1, 3) with generator
vector. sequences mentioned in the above
(iii) By using the time domain question 3.
approach find the output sequence
5. For the encoder (2, 1, 2) construct
2. Repeat the above question and find the viterbi decoding algorithm for
the output code vector using the received sequence R= [11 01 10
Transform Domain approach. List 01 00].
the distinct advantages of cyclic 6. Explain the operation of the
codes over linear block codes. convolution code with an example.
3. For the (2, 1, 3) convolution encoder 7. For the (3, 1, 3) convolution encoder
with the generator sequences with with the generator sequences with
the code rate r =1/2 and constraint the code rate r =1/2 and constraint
length K=4 are length K=4 are g(1) =[1 0 1 1] g(1) =[1
1 1 1] and g(2) =[1 1 0 1].
g(1) =[1 1 1 1] and g(2) =[1 1 0 1]
(i) Draw the block diagram of
(i) Draw the block diagram of encoder.
encoder.
(ii)For the message sequence m= {1
(ii)For the message sequence m= {1
1 1 0 1 1 0 1} construct the generator
0 0 0 1}. Construct the generator matrix and the find the output code
matrix and the find the output code vector.
vector.
321
(iii)Using the time domain approach

find the output sequence.
(iv)Verify the result by transform
domain approach.
322
Convolution codes
MULTIPLE CHOICE
QUESTIONS
1. In Frame Check Sequence, which horizontal axis represent/s?
code is used if character length is 6 a. Continuous time
bit and generates 12 bit parity b. Discrete time
check bits? c. Sampled time
a. CRC-12 d. All of the above
b. CRC-16 6. For the 4 states of an encoder on
c. CRC-32 vertical axis of Trellis diagram, what
d. CRC-CCITT do/does the solid line indicate/s?
2. Decoding of RS code comprises the a. ‘0’ input
determination of error …….. b. „1‟ input
a. Position c. Both „0‟ and „1‟
b. Angle d. None of the above
c. Position and magnitude
7. To obtain the transfer function of a
d. Frequency
convolution code, the splitting of
3. In RS code, the length is…….less all-zero state takes place into………
than number of symbols in symbol
a. Starting state
set (q).
b. First return to all zero state
a. One c. in-between state
b. Two d. Both a and b
c. Three
8. The distance between each symbol
d. Infinite
in given sequence and reference
4. In Minimum Distance Separable sequence is known as ………………
codes, the minimum distance is one
a. Euclidean Distance
more than the number of ………..
b. Distance between sequences
a. Parity check bits c. Manmattan Distance
b. Information bits d. Hamming Distance
c. Symbol bits 9. The term ‘dfree‟ is defined as the
d. None of the above Euclidean distance of coded signal
5. In Trellis diagram, what do/does the
323
in terms of ………possible distance 10.For designing trellis code, the

between all allowed sequences. emphasis must be on
a. smallest maximizing…………
b. largest a. Euclidean distance between code
c. average vectors
d. constant b. Hamming distance of error
correcting codes
c. Both a and b
d. None of the above
Answers
Q. No 1 2 3 4 5
Options a c a a b
Q. No 6 7 8 9 10
Options a d b a a
References
[1] R. Johannesson and K. S. Zigangirov, Fundamentals of Convolution Coding, 2nd
Edition, Wiley‐IEEE Press, 2015.
[2] E. Biglieri, D. Divsalar, P. J. McLane and M. K. Simon, Introduction to Trellis Coded
Modulation with Applications, Macmillan, 1991.
and Sons: 1989.
[4] T. K. Moon, Error Control Coding: Mathematical Methods and Algorithm, John Wiley &
Sons: New Jersey; 2005
[5] Shanmugam K San, Digital and Analog Communication System, John Wiley & Sons:
New York; 1989.
[6] H Meliani and A Benidir, Coding and Decoding Algorithms for BCH Codes, Al-Asha
College of Technology, KSA University of Blinda: Algeria.
324
Convolution codes
[7] Shuin and Daniel J Costello, Error Control Coding: Fundamentals and Applications,
Prentice Hall Series, Inc: Englewood Cliffs, NJ; 1983.
Delhi; 2008.
[9] Peter Sweeney, Error Control Coding, John Wiley and Sons: New York; 1991.
[11] J S Chitode, Information Theory and Coding, Technical Publications Pune, 2012.
[12] K Giridhar, Information Theory and Coding, Pooja Publications.
325
Work Space
326
Appendix-A
INFORMATION THEORY AND CODING
MODEL QUESTION PAPER – 1 (CBCS)
As per VTU Syllabus
Semester: V Max. Marks: 80

Code: 15EC55 Time: 3 Hours
Note: Answer any FIVE full questions, choosing one full question from
each module.
Module-1
Q. No 1 a. For the given Markov source shown in fig Q. 1(a).
Fig. Q. 1(a)
Determine
(i) State Probabilities (ii) Entropy of each source (iii) Entropy of source iv) G1>G2>H(s).
(8 Marks)
b. A black and white TV picture consists of 256 lines of picture information. Assume that
each line is consists of 256 picture elements (pixels) and that each can have 255 brightness
levels. Picture is repeated at the rate of thirty frames/sec. Calculate the average rate of
information conveyed by a TV set to a viewer.
(4 Marks)
c. Define the following terms
(i) Self information (ii) Entropy (iii) Rate of information iv) Mutual information
(4 Marks)
327
Model Question Paper 1
OR
Q. No 2a. The binary source produces symbols ‘0’ and ‘1’ with a probability p and 1-p.
Determine the Entropy of this source and sketch the variations of the entropy
with p.
(5 Marks)
b. The output of an information source consists of 128 symbols. Sixteen of which
occurs with probability of occurs with the probability of . The source emits
1000 symbols per second. Assume that the symbols are chosen independently.
Find the average information rate of the source.
(5 Marks)
c. For the given Markov source shown in fig. Q. 2(c).
Fig. Q. 2(c)
Determine (i) Entropy of source (ii) Show that G1>G2>H(s). (6 Marks)
Module-2
Q. No 3 a. A source emits symbols with probabilities 0.4, 0.2, 0.12, 0.08, 0.08, 0.08, 0.04.
Construct a binary Huffman code and Shannon-Fano code. Calculate the
efficiency in both the cases.
(8 Marks)
b. Explain Shannon’s noiseless encoding algorithm.
(4 Marks)
c. State all the properties of entropy and prove the Extremal property.
(4 Marks)
OR
328
Q.No 4a. The source emits messages consisting of two symbols each. These messages and their
probabilities are given in table 1. Design the source encoder using Shannon’s
encoding algorithm and also find encoder efficiency.
Table 1. List of probabilities for the message symbols
Messages (Mi) AA AC CC CB CA BC BB
Probabilities (Pi) ⁄ ⁄ ⁄ ⁄ ⁄ ⁄ ⁄
(6 Marks)
b. Construct a Shannon Fano Ternary code for the following ensemble shown in table 2.
Calculate code efficiency and redundancy. Also draw the corresponding code tree.
Table 2. List of probabilities for the message symbols
Symbols (S) S1 S2 S3 S4 S5 S6 S7
Probabilities (Pi) 0.3 0.3 0.12 0.12 0.06 0.06 0.04
With x={0, 1, 2}.

(6 Marks)
c. Obtain the code-words using Lempel-Ziv coding method for the given string
XXYYXXYYYXY.
(4 Marks)
Module-3
Q. No 5 a. For a given channel matrix, calculate H(x), H(y) and channel capacity given
P(x1)=0.6, P(x2)=0.3 and P(x3)=0.1
⁄ ⁄
⁄ ⁄
[ ⁄ ⁄ ]
(6 Marks)
b. Determine Channel capacity of a binary erasure channel.

(6 Marks)
c. Show that H(x, y)=H( ) + H(y).
329
(4 Marks)
OR
Q.No 6a. The joint probability matrix of a channel is given below.
[ ]
Compute H(x), H(y), H(xy), H( ⁄ ) and H( ⁄ ).

(6 Marks)
b. Show that H(x, y)=H( + H(y) and H(xy)=H(x)+ H( ⁄ ).
(4 Marks)
c. A non symmetrical binary channel is shown in fig Q. 3(c)
Fig. Q. 3(c)
Where P[x=0]= and P[x=1]= . Determine I[X, Y] for =0.75 and =0.9 and also find the
capacity of channel for rs=1000 sym/sec.
(6 Marks)
Module-4
Q. No 7 a. For a systematic (6, 3) linear block code the parity matrix given by
330
[P]=[ ]
(i) Find all the possible code vectors.

(ii) Find the minimum weight of the code.
(iii) Find the parity check matrix.
(iv) For a received code vector R=[ 1 1 0 0 1 0 ] detect and correct the error that has
occurred due to noise.
(6 Marks)
b. What are different methods of controlling errors? Explain.

(2 Marks)
c. The parity check bits of a (7, 4) Hamming codes are generated by
C5=d1+d3+d4
C6= d1+d2+d3
C7=d2+d3+d4
Where d1, d2, d3 and d4 are message bits.
(i) Find generator matrix [G] and parity check matrix [H] for this code.
(ii) Prove that GHT=0
(iii) Find the minimum weight of this code.
(iv) Find error detecting and correcting capability.
(v) Draw encoder circuit and syndrome circuit for the same.
(8 Marks)
OR
Q.No 8a. A (15, 5) linear cyclic code has a generator polynomial g(x)=1+x+x 2+x4+x5+x8+x10.
Draw the cyclic encoder and find the codeword for the message polynomial
D(x)=1+x2+x4 in systematic form by listing the states of the shift register. Also draw
the syndrome calculator circuit for given g(x).
(8 Marks)
b. What is binary cyclic code? Describe the features of encoder and decoder used for
cyclic codes using an (n-k) bit shift register.
(8 Marks)
Module-5
Q. No 9 a. Write a note on Galois field, Golay code, RS code and BCH code.
(8 Marks)
331
b. Consider the (3, 1, 2) convolution code with g(1)=(1 1 0), g(2)=(1 0 1) and g(3)=(1 1 1).
(i) Draw the encoder block diagram.
(ii) Find the generator matrix.
Find the codeword corresponding to the information sequence (1 1 1 0 1) using time

domain and transform domain approach.
(8 Marks)
OR
Q.No 10a. Consider the (2, 1, 3) convolution encoder with g (1)=(1 1 0), g(2)=(1 1 0 1) and
g(3)=(1 0 1 1).
(ii) Write down the state transition table.
(iii) Draw the code tree.
(iv) Find the encoder output produced by the message sequence (1 1 1 0 1) by
traversing through code tree.
(12 Marks)
b. Explain BCH Code with a suitable example.
(4 Marks)
332
INFORMATION THEORY AND CODING
MODEL QUESTION PAPER-2 (CBCS)
As per VTU Syllabus
Semester: V Max. Marks: 80

Code: 15EC55 Time: 3 Hours
Note: Answer any FIVE full questions, choosing one full question from
each module.
Module-1
Q. No 1 a. A zero memory source emits one of the four symbols in every 1ms. The symbol
probabilities
are , -. Find
(i) The average information content of the source in bits/symbol, trinits/symbol,

Hartley/symbol and nats/symbol.
(ii) Average information rate of the source.
(6 Marks)
b. A code is composed of dots and dashes. Assume that a dash is 3times as long as a dot
and has one-third the probability of occurrence. Find
(i) The average information content in a dot and dash
(ii) The entropy of dot - dash code
(iii) The average rate of the information, if a dot lasts for 10ms and this time is
allowed between symbols.
(8 Marks)
c. If there are „M’ equally likely and independent messages, then prove that amount of
information carried by each message will be of Ik=N bits, where M=2N and N is an
integer.
(2 Marks)
OR
333
Q. No 2a. In a facsimile transmission of picture there are about 2 x106 pixels/frame. It is

necessary to have eight brightness levels during good reproduction. Assume that
all the levels are equally likely to each other. Calculate the rate of information if
one picture is transmitted for every 4 seconds. Also, find the source efficiency of
this facsimile transmitter.
(5 Marks)
b. Consider a source S emits one of the three symbols S0, S1 and S3 with their
probabilities , and respectively. Find the entropy of source and list all the
elements for the second extension of this source. Also, Verify H (s2) = 2 H(s).
(5 Marks)
c. For the Markov source shown in fig 2(c), find
(ii) State entropies
(iii) Source entropy.
Fig. 2(c): Markov Source

(6 Marks)
Module-2
Q. No 3 a. Apply Arithmetic Coding procedure to generate a tag that lies in between 0 to 1 for
the probabilities P(V)=0.4, P(T)=0.3, P(U)=0.2, P(E)=0.08, P(C)=0.02. Also find the
total tag value for the sequence „VTUEC‟.
(6 Marks)
b. Consider a discrete memory less source S={S1, S2, S3, S4, S5, S6} with the probabilities
P={ 0.4, 0.2, 0.1, 0.1, 0.1, 0.05, 0.05}. Construct a binary Huffman code by placing the
composite symbol as high as possible. Also calculate the variance of word length.
(6 Marks)
c. Encode the given string 1001101001 using Lempel-Ziv-Welsh algorithm
(4 Marks)
OR
334
Q.No 4a. A discrete memory-less source with alphabets “A to H” has respective probabilities
0.22, 0.20, 0.18, 0.15, 0.10, 0.08, 0.05 and 0.02. Construct binary code and ternary
codes for the same using Huffman‟s encoding algorithm. Determine code efficiency
in each case.
(6 Marks)
b. Explain briefly the Lempel-Ziv and Lempel-Ziv Welsh coding technique.
(6 Marks)
c. Explain the important properties of codes to be considered while encoding a source.

(4 Marks)
Module-3
Q. No 5 a. Consider a set of symbols A= {a1, a2, a3} at the transmitter and a set of symbols B=
{b1, b2, b3} at the reception. For the table 1, find the missing probabilities (*) and also
find P( ⁄ ) and P( ⁄ ).
Table 1: Probabilities of symbols

b1 b2 b3
a1 0.7 0.1 *
a2 0.3 * 0.2
a3 * 0.4 0.6
bj 1.0 * 1.0
(6 Marks)
b. Two noisy channels are cascaded whose channel matrices are given by
( ⁄ )=* +
and
( ⁄ )=[ ]
With P(X1)=P(X2)=0.5. Also show that I (X, Y) > I (X, Z).

(6 Marks)
c. Define Communication channel. With an example briefly explain the representation
and matrix of channel.
(4 Marks)
OR
335
Q.No 6a. The noise matrix of a binary symmetric channel is given below, with the transmitted
symbols having probabilities of and respectively.
P( ⁄ )=* +
(i) Calculate H(X), H(Y), H(X, Y), H( ⁄ ), H( ⁄ ) and I(X, Y)

(ii) Determine the Capacity of Channel „C‟.
(iii) Find the Efficiency and redundancy of the channel.
(8 Marks)
b. Calculate the capacity of the channel using Muroga‟s method for the fig Q. 6(c) and
P(X1)=0.5, P(X2)=0.3, and P(X3)=0.2 .
Fig. Q 6(c): Channel diagram

(8 Marks)
Module-4
Q. No 7 a. For the systematic (7, 4) linear block code. Calculate
(i) The Valid code vectors for the parity matrix [P] given below.
[P]=[ ]
(ii) Construct the standard array.

(iii) Find the status of error detecting and error correcting capabilities of code.
Draw the corresponding encoding and decoding circuit.
(8 Marks)
336
b. Consider a (6, 3) linear code whose generator matrix is
[G]=[ ]
(i) Find all possible valid code vectors.

(ii) Find all the hamming weights and distances.
(iii) Find minimum weight parity check matrix.
(iv) Draw the corresponding encoding circuit.
(8 Marks)
OR
Q.No 8a. Consider (15, 11) cyclic code generated by g(x)=1+x+x4.
(i) Device a feedback registers encoder for this code.
(ii) Illustrate the encoding procedure with message vector 11001101011 by listing
the states of the register.
(8 Marks)
b. For n 7, identify (n, k) values of the cyclic codes generated by the following
generator polynomials.
(i) g(x)=1+x2+x3
(ii) g(x)=1+x+x2+x4
(iii) g(x)=1+x2+x3+x4
(8 Marks)
Module-5
Q. No 9 a. Write a note on Galois field, RS code and BCH code.
(4 Marks)
b. Consider the binary convolution encoder shown in fig. Q 9(b). Draw the state table,
state transition table, state diagram and the corresponding code tree. Using the
code tree, find the encoded sequence for the message (1 0 1 1 1). Verify the output
sequence so obtained using the transform domain approach.
(12 Marks)
337
Fig. Q 9(b)
OR
Q.No 10a. Consider the (3, 1, 2) convolution encoder with g(1)=(1 1 0), g(2)=(1 0 1) and
g(3)=(1 1 1).

(ii) Write down the generator matrix and output for 1 1 1 0 1
(iii) Find code word for 1 1 1 0 1 using time domain method.
(iv) Draw state diagram and tree diagram.
(12 Marks)
b. Explain Golay Code with a suitable example.
(4 Marks)
338
INDEX Digital communication
Discrete channel
Discrete communication channels
A Discrete memory less source
Average rate of information
Average information content per symbol E
Average amount of information per symbol
Efficiency of channel
Arithmetic coding
Encoding cyclic codes
Algorithm
Ensemble
Algebraic description of cyclic codes
Entropy
Automatic Repeat request
Equally likely events
Euclidean distance
B EXOR Operation
Binary symmetric channel (BSC) Equivocation
Binary Erasure Channel (BEC) Error control coding
Block codes Extended Huffman coding
Bose -Chaudhuri-Hocquenghen (BCH) codes
Branch Metric F
Burst Errors
Fixed length coding
Burst Error correction Capability
Factorial
Baye’s rule
Finite field
Binary code
Fixed length coding
Binary encoder
Binary Golay codes
G
Bir error rate (BER)
Generator
Galois field
C Generator sequence
Channel Capacity
Generator matrix
Convolution Encoder
Generator polynomial
Channel Diagram
Golay codes
Channel Transition Probability
Codeword H
Channel matrix Hamming codes
Continuous channel Hamming bound
Communication system Hamming distance
Convolution encoding Hamming weight
Constraint length Huffman codes
Cyclic codes
Code tree
I
Code vectors
Identity element
Compression
Information channels
Information rate
D Information source
Decoding of Convolution code Instantaneous codes
Division of code tree Information conveyed by event
339
Index
Information redundancy O
Irreducible polynomial Optimal codes
Outcome
J Operator
Joint Probabilities Output symbol probabilities
Joint probability
Joint occurrence of events P
Joint Probability Matrix Parity bits
Parity check matrix
K Periodic channel
K-bit message symbol Properties of probability
Kraft-McMillan inequality Perfect codes
Posteriori entropy
L Prefix codes
Lempel –Ziv algorithm Primitive element
Linear block codes Probability of an event
Lower bound entropy Probability of errors in the channel
Lempel –Ziv-Welsh algorithm Product codes
Linear feednack shift registers
Linearly independent vectors Q
Lossless compression Q-array BCH codes
Lossy compression
Lossy compression algorithms R
Reed Solomon (RS) codes
M Redundancy of the source
Markoff model Relative entropy
Markoff source
Maximum average information S
Multiplicative Inverse Self Information
Maximum entropy Shannon-Fano Algorithm
Minimum Hamming distance Shannon- Hartley Theorem
Muroga’s theorem Shannon- channel coding theorem
Mutual information Shannon first encoding theorem
Memory element Shifting register
Signal to noise ratio
N Source coding theorem
‘n’ bit codeword Source efficiency
(n-k) dimensional subspace Source encoder
noise matrix Source entropy
non-singular codes State transition table
non-systematic cyclic codes State diagram
nth order extension of source Symbol probability
nth order Markoff source Surviving path
Syndrome calculation
Second order extension of source
340
Index
Systematic Cyclic codes

Subclass of linear block codes
T
Tail of the tree
Time domain approach
Transform domain approach
Trellis diagram
Turbo codes
U
Upper bound entropy
Uncertainty
Uniquely decodable codes
Units
Uniform distribution
V
Valid codewords
Variance
Viterbi algorithm
Vectors
Viterbi decoder
X
XOR operation
Z
Zero memory sources
Zero memory source extension
341

Information Theory and Coding Full Book

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Information Theory and Coding Full Book

Uploaded by

Copyright:

Available Formats

INTRODUCTION TO

Prof. SUNIL S HARAKANNANAVAR,B.E., M. Tech., (Ph.D.,)

Prof. Veena I Puranikmath, B.E., M. Tech., (Ph.D.,)

Dr. Prashanth C R, B.E., M. Tech., Ph.D.

Author : Prof. Sunil S Harakannanavar, Prof. Veena I Puranikmath

CopyRight :Prof. Sunil S Harakannanavar

First Edition : July 2018

All rights reserved. No part of this publication can be reproduced in any

Features of the Book

Part one: Information Theory: Introduction, Measure of information, Information

RBT Level: (L1, L2, L3)

Part Three: Information Channels: Communication Channels. Channel Models,

Part Seven:Convolution codes:Convolution Codes: Convolution Encoder, Time

RBT Level: (L1, L2, L3)

How to Use the Book

The book is intended for undergraduate and postgraduate students of Electronics

Suggestions for improvement of the book are welcome.

Prof. Sunil S Harakannanavar completed his Bachelor of Engineering

Prof. Veena I. Puranikamth graduated in the field of Electronics and

Dr. Prashanth C R received the B.E degree in Electronics, ME degree

2.2.6 Optimal Codes............................................................................................ 60

3.5.5 Deterministic Channel................................................................................. 165

4. Error Control Coding

5. Binary Cyclic Codes

5.4 Encoding using (n-k) bit shift registers................................................................ 240

6. Some Important Cyclic Codes

7.5 Analysis of convolution encoder using transform domain approach..............290

Appendix A: Model Question Papers

Syllabus: Introduction, Measure of information, Information content of message,

1. Information Theory Page. No

After reading this chapter, you will be able to:

1.2 Basics of Information System

Figure 1.1: Block diagram of an Information System

1.3 Measure of Information

Amount of Information (Ik)= log2

1.3.2 Properties of Information

i) If there is more uncertainty about the message, information is also carried

I (S1, S2) = I1 +I2

Solution: We have, the amount of information is given by,

The individual information carried by both messages m1 and m2 are

I1= log2 and I2 = log2

Hence the information carried by ‘m1’ and ‘m2’ is

I1, 2= log2 = (Ik) = log2* +

I1, 2= {log2 + log2 }

Solution: We have, Self information content of a symbol can be calculated using

I0= log2 = log2 = = 2 bits

I1= log2 = log2 = = 0.415 bits

Therefore, the total information conveyed is, I = I0 + I1 = (2+0.415)

Solution: (i) For Bits, We have,

Is0= log2 = log2 ⁄

Is1= log2 = log2 ⁄

Is0= log3 = log3 = 0.63 trinits

Is1= log3 = log3 ⁄

Is0= log10 = log10 ⁄

Is1= log10 = log10 ⁄

Is0= loge = loge ⁄

Is1= loge = loge ⁄

EXAMPLE 1.6 Comment on the information content of the following messages

1.3.3 Average Information Content (ENTROPY) of Symbols in Long

Now, the self information of ‘S1’ is given by

The total information conveyed by symbol ‘S2’ is

P2L log2 bits

Similarly, the total information conveyed by symbol ‘S3’ is

P3L log2 bits