HETT 202 – Group 2

Presentation
Topics :
1) Source Coding & Theorems,
2) Prefix Coding & Huffman Codes,
3) Mutual Information & Channel Capacity Concepts
 Source Coding & Theorems

• Source coding is the first step of processing input signals.

• It identifies, quantifies and converts the analog signals to digital
formats and then uses simpler and smaller codes to represent the
information carrying signal.
• It achieves this in 2 ways:
1. It can remove the redundant or unimportant extra information
in the source signal. This is “lossless” coding.
2. It can also remove some parts or portions of the source signal
which are important and lose some information in the process.
This is called “lossy” coding.
Why is it done?

• The main purpose of source coding is to represent the source signal at

a lower rate e.g by compression.
• There are a number of reasons why this done but there 2 main reasons
that we are going to consider:
1. To represent the source signal with the minimum number of bits
possible without loss. This reduces the risks of noise and ISI etc
which increase as the rate of transmission increases.
2. The channel may not have the capacity to communicate/transmit at
the source information rate which will cause congestion in the
system. Source coding then reduces the number of bits that is used
to represent the signal thereby reducing the number of bits that are
transmitted.
General Overview

• At the receiver side the symbols have to be guessed based on statistical

communications theory.
• To get a more accurate guess and get around the errors associated with
source coding we employ several techniques. E.g
1. We can employ a method which gives the receiver the ability to
detect an error. This uses ARQ (Automatic Repeat Request) methods
to tell the source to re-transmit that portion which has an error.
2. We can also employ a method which involves adding enough
redundant symbols to the information symbols, to allow the capability
of detecting and also correcting the wrongly received symbols. It uses
FEC (Forward Error Correction).
Among others.
“Lossless” Coding

• This describes the algorithms that allow the exact reconstruction

of the original source data from the compressed data.
• This is sometimes referred to as noiseless or entropy coding.
• It can only be employed to Discrete time signals and Discrete
amplitude signals
• Can only achieve moderate compression
• Used in “zip”
“Lossy” Coding

• These describe algorithms that are characterised by an irreversible

loss of information.
• An approximation is reconstructed from the compressed data.
• This achieves relevantly higher bit rate reduction than with
lossless coding.
• Used in “JPEG” and “MPEG
Discrete Memoryless Source

• In source coding we use a Discrete Memoryless Source.

• A discrete memoryless source generates symbols in discrete time
with each of the symbols being a random variable and is
independent of the previous symbol.
• A DMS can be a sampled and then quantized analog source. It is
sampled to make it discrete and quantized to make the set of
values finite. Memoryless means each symbol is uncorrelated from
the other.
Source Coding Theorems

1. It states that the average length of a code(L) can never

be greater than the entropy of a source(H(S)).
• In mathematical terms log2L ≥ H(S)
• This indicates how far we can compress the code (by removing
minimising/removing redundancy) for conveying source
messages.
• This is also known as noiseless coding theorem
2. Let a discrete channel have the capacity C and a discrete source
the entropy per second H. If H≤C there exist a coding system
such that the output of the source can be transmitted over the
channel with an arbitrarily small frequency of errors. AND if H>C
it is possible to encode the source so that the equivocation is
less than H–C+ε.
• This theorem shows us the redundancy the system could use in
order to facilitate error correction.
Terms to know and remember

• Bandwidth is the frequency range that a communication channel

can transmit without any significant loss
• Channel capacity is the maximum data that can be transmitted
through a channel without any significant loss
• Bit rate is the number of bits transmitted per unit time
• AGWN acronym for additive white Gaussian noise
Shannon –Hartley Theorem

• We can not talk about channel capacity and not mention

Shannon –Hartley theorem
• It states that ;
If transmission rate (R) is less than channel
capacity(C) then the data transmission can
be made with arbitrarily small error
probabilities using the coding techniques
Shannon –Hartley Equation

• The equation relates max capacity(bit rate) that can be achieved

given a certain channel with certain noise characteristics (AWGN)
and bandwidth
• C = B Log2 (1 + S/N) ,
• C= max capacity(bits/sec)
• B= bandwidth of channel (Hz)
• S= signal power (W)
• N =noise power (W) additive in nature
EXAMPLE

• Calculate the channel capacity of a medium whose bandwidth

handles a signal in the range 820MHz to 900MHz .At that range of
frequency the SNR is 20dB ?
Factors that affect channel capacity

• From the equation we can tell that channel capacity is affected by

bandwidth and SNR
• Bandwidth limits how fast the information symbols can be send
through a channel
• SNR limit how much info we can squeeze in each transmitted
symbol……Increasing SNR makes the signal more robust against noise
• Other factors also include the type of channel and the distance
between the transmitter and the receiver

• In summary the Shannon –Hartley theorem helps us to maximize the

channel using reasonable amount of bandwidth with a better signal
SNR to improve efficiency and quality
Mutual Information

• In communications, mutual information in its most basic

definition, is the amount of information transmitted through a
noisy channel.
• Mutual information is the average amount of information that you
get about x from observing the value of y.
• It is the amount of information one random variable contains
about another.
• It reduces the uncertainty of x due to knowledge of y & vice versa.
• It is the measure of independence where :
• I(X;Y) = 0 , where the two variables are independent
• It grows according to
• The degree of dependence
• The entropy of the variables.
Mathematical Representation / Equations

• The defining equation :

Graphical Representation

• The graphical relationship of data stream x, & data stream y :

Forms / Types of Mutual Information
Prefix Coding

Definition of terms:
coding- assignment of binary sequences to the alphabet.
code- set of binary sequence.
codeword- individual member of the code.
alphabet- collection of letters or symbols.
Example
code word
a1 - 0
a2 - 1
alphabet a3 - 10 code
a4 - 11
• any variable length encoding scheme(a coding scheme which uses
code words of varying lengths for encoding the various symbols)
that has the important property that no code word is a prefix of
another.
• eg In a prefix code, if the character 'a' is encoded as 0 then no
other character has a code that starts with 0.
Prefix:
a→010
b→01011
we can say that 'a' is a prefix of 'b' because 010 of 'a' marks the start
of 01011 of 'b'. 010 is the prefix of 'b' and 11 is the dangling suffix
because it does not match.
Desirable Properties of Symbol Codes:
1. efficiency→getting good compression.
2. speed→fast so that we encode and decode quickly.
3. simplicity→ understand easily and implement easily in algorithm.
Determining if a code is a prefix code or not:
1. a1→0 2. a1 → 0
a2→1 a2 → 10
a3→10 a3 → 110
a4→11 a4 → 111
• if all letters are at the external node of the tree then it is a prefix code.
• external nodes are leaf nodes and internal nodes are not leaf nodes.
• if a cdeword is a prefix of another code word then it would be on the
path to the code word required
Uniquely Decodable Code

• sequence of codes that can be decoded in only one way.

Example 1:
a1 → 0 a2a1
a2 → 1 10
a3 → 10 a3
a4 → 11
• we can now conclude that the sequence of codes cannot be
decoded in only one way hence it is not uniquely decodable.
Example 2:
a1 → 0
a2 → 10 100 a2a1
a3 → 110
a4 → 111
Algorithm to test UDC

1. group the code words.

2. if no prefix found
then it is UDC
else if prefix found
then add dangling suffix to the set.
3. if dangling suffix = some code word = not UDC
else if all dangling suffix are over = UDC
Right Prefix
0
10
110, this code has a right prefix.
• perhaps you're asked to determine if a code is UD and one might say it's difficult to discern, but if
you check the right prefix then you know it's uniquely decodable.
Huffman Coding:

• used for compression.

• used only when the probability of element source file is
known(minimum huffman coding), if it's not known then we
use adaptive huffman coding.
• when data is sent over a network, data can be compressed
and sent to reduce the cost of transmission.
Message: BCCABBDDAECCBBADDCC
LENGTH 20
• message has to be sent using ASCII code.
• ASCII codes are 8 bits.
Finding the code word for letters using huffman coding:
• we generally have two methods that answer the question
Method 1:
1. arrange letters in descending order pertaining to their probabilities.
2. take two letters with the least probabiities and merge them, copy others
as is. If sum is the same as the some other already present probabilities
then we will put down other probabilities
3. add a zero and one on the merged probabilities
4. repeat step 2
5. now we will find the code word for the letters (from right to left)
6. find number of bits per symbol required.
Method 2:
1. construct a tree using letters with the least probabilities.
2. if sum = next element
then we put the next element to the left
if sum < next element
then we put the next element to the left
if sum > next element
then we put the next element to the right
3. we go up front from the bottom
NB: Letters having the highest probability will have smaller codewords, also
the length of code word of two least probabilities are the same.