TC-502 INFORMATION THEORY

SOURCE CODING
 The Kraft Inequality
If the individual code word lengths are specified there is no
guarantee that an instantaneous code can be designed with
those lengths.
The Kraft Inequality theorem provides a limitation on the
code word lengths for the design of instantaneous codes.
 The Kraft Inequality
(Binary case, i.e. r=2)
 McMillans Theorem
The use of shorter code word lengths creates more efficient
codes. Since the class of uniquely decodable codes is larger
than the class of instantaneous codes, one would expect
greater efficiencies to be achieved considering the class of all
uniquely decodable codes rather than the more restrictive
class of instantaneous codes. However, instantaneous codes
are preferred over uniquely decodable codes given that
instantaneous codes are easier to analyse, systematic to
design and can be decoded using a decoding tree (state
machine) structure.
McMillans Theorem assures us that we do not lose out if we
only consider the class of instantaneous codes. If the
individual code word lengths are specified there is no
guarantee that an instantaneous code can be designed with
those lengths.
McMillans Theorem
Code Length and Code Efficiency
Source Coding Theorem
PROBLEM
 ENTROPY CODING
The design of variable-length code such that its
average code word length approaches the entropy of
the DMS is referred to as entropy coding.
Two common examples of entropy coding
Shannon-Fano Coding
Huffman Coding
Shannon-Fano Coding
Example
Huffman Encoding
Example
PROBLEM