Learning Material - An Introduction To Digital Communication by Texas A&m University

An Introduction to
Digital Communications
Costas N. Georghiades
Electrical Engineering Department
Texas A&M University
These
Thesenotes
notesare
aremade
madeavailable
availablefor
forstudents
studentsof
ofEE
EE455,
455,and
andthey
theyare
areto
to
be
used
to
enhance
understanding
of
the
course.
Any
unauthorized
be used to enhance understanding of the course. Any unauthorized
copy
copyand
anddistribution
distributionof
ofthese
thesenotes
notesisisprohibited.
prohibited.
Course Outline
Introduction
Analog Vs. Digital Communication Systems
A General Communication System
Some
Probability Theory
Probability space, random variables, density
functions, independence
Expectation, conditional expectation, Bayes rule
Stochastic processes, autocorrelation function,
stationarity, spectral density

Outline (contd)
Analog-to-digital
conversion
Sampling (ideal, natural, sample-and-hold)

Quantization, PCM
Source
coding (data compression)
Measuring information, entropy, the source coding
theorem
Huffman coding, Run-length coding, Lempel-Ziv
Communication
channels
Bandlimited channels
The AWGN channel, fading channels
Outline (contd)
Receiver
design
General binary and
M-ary signaling
Maximum-likelihood receivers
Performance in an AWGN channel
RThe Chernoff and union/Chernoff bounds
RSimulation techniques
Signal spaces
Modulation: PAM, QAM, PSK, DPSK, coherent FSK,
incoherent FSK
Outline (contd)
Channel
coding
Block codes, hard and soft-decision decoding,
performance
Convolutional codes, the Viterbi algorithm,
performance bounds
Trellis-coded modulation (TCM)
Signaling
through bandlimited channels
ISI, Nyquist pulses, sequence estimation, partial
response signaling
Equalization
Outline (contd)
Signaling
through fading channels
Rayleigh fading, optimum receiver, performance

Interleaving
Synchronization
Symbol synchronization
Frame synchronization
Carrier synchronization
Introduction
A General Communication System
Source
Transmitter
Channel
Receiver
User
Source:
Speech, Video, etc.
Transmitter: Conveys information
Channel:
Invariably distorts signals
Receiver:
Extracts information signal
User:
Utilizes information
Digital vs. Analog Communication

systems have an alphabet which is
uncountably infinite.
Analog
Example: Analog Amplitude Modulation (AM)
Receiver
RF
Oscillator
Analog vs. Digital (contd)

r
Digital systems transmit signals from a
discrete alphabet
Example:
Binary digital communication systems
Data Rate=1/T bits/s
1
0110010...
or
Transmitter
Digital systems are resistant to noise...

1
Noise
s1(t)
s2(t) 0
r(t)
1?
Channel
?
Optimum (Correlation) Receiver:

r(t)
( )dt
0
s1(t)
t=T
>
<
1
0
0
Comparator
10
Advantages of Digital Systems

Error correction/detection
Better encryption algorithms
More reliable data processing
Easily reproducible designs
Reduced cost
Easier data multiplexing

Facilitate data compression
11
A General Digital Communication System

Source
A/D
Conversion
Source
Encoder
Channel
Encoder
Modulator
C
h
a
n
n
e
l
Synchronization
User
D/A
Conversion
Source
Decoder
Channel
Decoder
Demodulator
12
Some Probability Theory

Definition: A non-empty collection of subsets = {A1 , A2 , ...} of
a set , i.e., = {Ai ; Ai } is called an algebra of sets if:

Aj Ai Aj
1)
Ai and
2)
Ai Ai
Example: Let = {0,1,2} .
1) = {, } an algebra
2) = { , , {1} , {2} , {0} , {1,2} , {1,0} , { 0,2}} an algebra
3) = { , , {0} , {1} , {2}}
not an algebra
13
Probability Measure
Definition: A class of subsets, , of a space is a -algebra
(or a Borel algebra) if:
1) Ai Ai .
2) Ai , i = 1,2,3,... U Ai .
i =1
Definition: Let be a -algebra of a space . A function P

that maps onto [0,1] is called a probability measure if:
1) P[ ] = 1
2) P[ A] 0
A .

3) P U Ai = P[ Ai ] for
i =1 i =1
Ai A j = , i j .
14
Probability Measure
Let = (the real line) and be the set of all intervals (x1, x2] in .
Also, define a real valued function f which maps such that:
1) f ( x) 0 x .
2)
f ( x)dx = 1.
Then:
] [
P { x ; x1 < x x 2 } = P ( x1 , x 2 ] =
x2
x1
f ( x )dx
is a valid probability measure.
15
Probability Space
The following conclusions can be drawn from the above definition:
1) P[ ] = 0
[ ]
2) P A = 1 P[ A]
( P ( A + A ) = P ( ) = 1 = P ( A ) + P ( A )) .
3) If A1 A2 P ( A1 ) P ( A2 )
4) P[ A1 A2 ] = P[ A1 ] + P[ A2 ] P[ A1 A2 ] .
Definition: Let be a space, be a -algebra of subsets of , and

P a probability measure on . Then the ordered triple ( , , P ) is a
probability space.
sample space
event space
P
probability measure
16
Random Variables and Density Functions

Definition: A real valued function X() that maps into the real line
is a random variable.
Notation: For simplicity, in the future we will refer to X() by X.
Definition: The distribution function of a random variable X is defined by
F X ( x ) = P[ X x ] = P[ < X x ] .
From the previous discussion, we can express the above probability in terms
of a non-negative function f X () such that
( x ) d X = 1 as follows
FX ( x ) = P [ X x ] =
( ) d .
We will refer to f X () as the density function of random variable X.

17
Density Functions
We have the following observations based on the above definitions:
1) F X ( ) =
f X (x)d X = 0
2) F X ( ) =
f X (x)d X = 1
3) If x 1 x 2 F X ( x 1 ) F X ( x 2 ) ( F X ( x ) non-decreasing)
Examples of density functions:
a) The Gaussian density function (Normal)
f X ( x) =
1
2
( x )2
2 2
18
Example Density Functions

b) Uniform in [0,1]
1, x [ 0,1]
f X (x) =
0, otherwise
fX (x)
1
x
0
c) The Laplacian density function:

fX (x)
f X (x) =
a
exp ( a x )
2
19
Conditional Probability
Let A and B be two events from the event space . Then, the probability of event A,
given that event B has occurred, P[A | B ] , is given by
P[ A B]
.
P[ A| B ] =
P[ B]
Example: Consider the tossing of a dice:
P [{2} | "even outcome"]=1/3,
P[{2} | "odd outcome"] = 0
Thus, conditioning can increase or decrease the probability of an event, compared to its
unconditioned value.
The Law of Total Probability
M
Let A1, A2,..., AN be a partition of , i.e.,
UA
i =1
= and Ai A j = i j .
Then, the probability of occurrence of event B can expressed as

M
P[ B] = P[ B| Ai ] P[ Ai ] , B .
i =1
20
Illustration, Law of Total Probability

P ( B | A3 )
A3
B
A1
A2
P ( B | A2 )
21
Example, Conditional Probability

Pr( 0) =
1
2
P00
P01
P00=P[receive 0 | 0 sent]
1
Pr(1) =
2
P10
1
P11
P01 = 0.01
P00 = 1 P01 = 0.99
P10 = 0.01
P11 = 1 P10 = 0.99
Pr( e) = Pr( 0) P01 + Pr(1) P10 =
1
1
0.01 + 0.01
2
2
= 0.01
22
Bayes Law
Bayes Law:
Let
Ai , i = 1, 2,..., M be a partition of and B an event in . Then
P Aj | B =
][ ]
P B| A j P A j
M
P[ B| A ] P[ A ]
i
i =1
Proof:
[
]
] P[ B] P[ A B] = P[ A | B] P[ B] = P[ B| A ] P[ A ]
P[ B| A ] P[ A ]
P[ B| A ] P[ A ]
P[ A | B] =
=
P Aj | B =
P Aj B
P( B )
P[ B| A ] P[ A ]
i =1
23
Statistical Independence of Events

Two events A and B are said to be statistically independent if
P[ A B] = P[ A] P[ B] .
In intuitive terms, two events are independent if the occurrence of one does not affect the
occurrence of the other, i.e.,
P ( A| B ) = P ( A )
when A and B are independent.
Example: Consider tossing a fair coin twice. Let
A={heads occurs in first tossing}
B={heads occurs in second tossing}.
Then
1
.
4
The assumption we made (which is reasonable in this case) is that the outcome of a coin
toss did not affect the other.
P[ A B ] = P[ A] P[ B] =
24
Expectation
Consider a random variable X with density fX (x). The expected (or mean) value of X is
given by
E[ X ] =
xf
( x )dx .
In general, the expected value of some function g( X ) of a random variable X is given by

E [ g( X ) ] =
g( x ) f
( x ) dx .
When g( X ) = X n for n = 0,1,2,L , the corresponding expectations are referred to as the

n-th moments of random variable X. The variance of a random variable X is given by
var ( X ) =
2
x
E
(
X
)
f X ( x )dx
[
]
f X ( x )dx E 2 ( X )
= E ( X 2 ) E 2 ( X ).
25
Example, Expectation
Example: Let X be Gaussian with
f X ( x) =
1
2 2
( x ) 2
exp
2
2
Then:
E( X ) =
1
2
xe
( x)2
2 2
dx = ,
Var( X ) = E [ X 2 ] E 2 ( X ) =
2
2
2
.
x
f
(
x
)
dx
26
Random Vectors
Definition: A random vector is a vector whose elements are random variables, i.e., if X1,
X2, ..., Xn are random variables, then
X = ( X 1 , X 2 ,..., X n )
is a random vector.
Random vectors can be described statistically by their joint density function
f X (x) = f X 1 X 2 ... X n ( x1 , x 2 ,L , x n ) .
Example: Consider tossing a coin twice. Let X1 be the random variable associated with
the outcome of the first toss, defined by
1, if heads
X1 =
0, if tails
Similarly, let X2 be the random variable associated with the second tossing defined as
1, if heads
X2 =
.
0, if tails
The vector X = ( X 1 , X 2 ) is a random vector.
27
Independence of Random Variables

Definition: Two random variables X and Y are independent if
f X ,Y ( x , y ) = f X ( x ) f Y ( y ) .
The definition can be extended to independence among an arbitrary number of random
variables, in which case their joint density function is the product of their marginal
density functions.
Definition: Two random variables X and Y are uncorrelated if
E [ XY ] = E [ X ] E [ Y ] .
It is easily seen that independence implies uncorrelatedness, but not necessarily the other
way around. Thus, independence is the stronger property.
28
The Characteristic Function

Definition: Let X be a random variable with density f X ( x ) . Then the characteristic
function of X is
X ( j ) = E e
j X
]= e
j x
f X ( x )dx .
Example: The characteristic function of a Gaussian random X variable having mean

and variance 2 is
X ( j ) =
1
2
j x
( x )2
2
dx = e
1
j 2 2
2
Definition: The moment-generating function of a random variable X is defined by

X ( s) = E [ e
sX
] = e
sx
f X ( x ) dx .
Fact: The moment-generating function of a random variable X can be used to obtain its
moments according to:
n
X ( s)
d
n
|s = 0
E[ X ] =
ds n
29
Stochastic Processes
A stochastic process {X (t ); < t < } is an ensemble of signals, each of which can be
realized (i.e. it can be observed) with a certain statistical probability. The value of a
stochastic process at any given time, say t1, (i.e., X(t1)) is a random variable.
Definition: A Gaussian stochastic process is one for which X(t) is a Gaussian random
variable for every time t.
5
Amplitude
Fast varying
0
-5
0
0.2
0.4
0.6
0.8
0.8
Time
1
Amplitude
Slow varying
0
-1
0
0.2
0.4
0.6
30
Characterization of Stochastic Processes
Consider a stochastic process { X ( ); < < } . The random variable X(t), t ,

has a density function f X ( t ) ( x; t ) . The mean and variance of X(t) are
E [ X ( t )] = X ( t ) =
xf
X (t )
( x; t )dx ,
VAR[ X ( t )] = E ( X ( t ) X ( t ) )
].
Example: Consider the Gaussian random process whose value X(t) at time t is a
Gaussian random variable having density
x2
f X ( x; t ) =
exp
,
2
t
2 t
We have E [ X ( t )] = 0 (zero-mean process), and var[ X ( t ) ] = t .

0.4
t= 1
0.3
0.2
0.1
0
-6
t= 2
-4
-2
31
Autocovariance and Autocorrelation

Definition: The autocovariance function of a random process X(t) is:
[(
)]
)(
C XX ( t1 , t 2 ) = E X ( t1 ) X ( t1 ) X ( t 2 ) X ( t 2 ) ,
t1 , t 2 . .
Definition: The autocorrelation function of a random process X(t) is defined by
R XX (t1 , t 2 ) = E X (t1 ) X (t 2 ) ,
t1 , t 2 .
Definition: A random process X is uncorrelated if for every pair (t1,t2)

E [ X ( t 1 ) X ( t 2 )] = E [ X ( t 1 )] E [ X ( t 2 )] .
Definition: A process X is mean-value stationary if its mean is not a function of time.

Definition: A random process X is correlation stationary if the autocorrelation function
R XX (t1 , t 2 ) is a function only of = (t1 t 2 ) .
Definition: A random process X is wide-sense stationary (W.S.S.) if it is both mean value
stationary and correlation stationary.
32
Spectral Density
Example: (Correlation stationary process)
X (t ) = a
RXX (t1 , t2 ) = exp t1 t2 = exp[
],
= t1 t 2 .
Definition: For a wide-sense stationary process we can define a spectral density, which is the Fourier transform of the
stochastic process's autocorrelation function:
SX ( f ) =
XX
( ) e j 2 f d .
The autocorrelation function is the inverse Fourier transform of the spectral density:
R XX ( ) =
S (f) e
j2 f
df .
Fact: For a zero mean process X,

var ( X ) = R XX (0) =
S ( f ) df .
X
33
Linear Filtering of Stochastic Signals

x(t )
H(f )
y (t )
SY ( f ) = S X ( f ) H ( f )
The spectral density at the output of a linear

filter is the product of the spectral density of the
input process and the magnitude square of the
filter transfer function
34
White Gaussian Noise

Definition: A stochastic process X is white Gaussian if:
a) X ( t ) = (constant)
N
b) R XX ( ) = 0 ( ) ( = t 1 t2 )
2
c) X is a Gaussian random variable and X(ti ) is independent of X(tj ) for all
ti t j .
Note: 1) A white Gaussian process is wide-sense stationary
N
2) S X ( f ) = 0 is not a function of f
2
Sx(f)
N0/2
f
35
Analog-to-Digital Conversion
r
Two steps:
Sampling
Discreetize amplitude: Quantization
Discreetize time:
A
m
p
l
i
t
u
d
e
Analog signal: Continuous time,

continuous amplitude
Time
36
Sampling
Signals are characterized by their frequency
content
The Fourier transform of a signal describes its

frequency content and determines its bandwidth
x(t)
0.3
X ( f ) = x (t )e j 2ft dt
Time, sec
x (t ) = X ( f )e
j 2ft
df
X(f)
0.5
Frequency, Hz
37
Ideal Sampling
Mathematically, the sampled version, xs(t), of signal x(t) is:
x s (t ) = h(t ) x (t ) X s ( f ) = H ( f ) X ( f ) ,
1
h (t ) = (t kTs ) =
Ts
k =
j 2 k
t
Ts
Sampling
function
k =
h(t)
...
-4Ts -3Ts -2Ts -Ts
...
0
Ts
2Ts 3Ts 4Ts
xs(t)
Ts
2 Ts 3 Ts 4 Ts
38
Ideal Sampling
1
H ( f ) =
Ts
j2
K =
kt
Ts
1
=
Ts
k =
k
.
Ts
Then:
1
X s ( f ) = H( f ) * X ( f ) =
Ts
.
X
f
K =
s
X s ( f)
Aliasing
fs < 2 W
X(f)
...
...
-fs
-W
fs
(b )
X s ( f)
No Aliasing
fs > 2 W
...
...
-fs
-W
fs
(a )
39
Ideal Sampling
If fs>2W, the original signal x(t) can be obtained from xs(t) through simple low-pass
filtering. In the frequency domain, we have
X ( f ) = X s ( f ) G ( f ),
where
Ts , f B
G( f ) =
0, oherwise.
for W B f s W .
G( f )
The impulse response of the low-pass filter, g(t), is then
g( t ) = 1 G( f ) = G( f ) e j 2 ft df = 2 BTs
B
sin(2 Bt )
.
2 Bt
Ts
B
B f
From the convolution property of the Fourier transform we have:

x(t ) =
x (a ) g (t a )da = x(kT ) (a kT )g (t a )da = x(kT ) g (t kT ) .

s
Thus, we have the following interpolation formula

x(t ) = x(kTs ) g (t kTs )
k
40
Ideal Sampling
G(f)
T
-B -W
W B
g(t)
t
The Sampling Theorem:
A bandlimited signal with no spectral components above W Hz can be recovered
uniquely from its samples taken every Ts seconds, provided that
Nyquist
1
Ts
, or, equivalent ly, f s 2W .
Rate
2W
Extraction of x(t) from its samples can be done by passing the sampled signal through a
low-pass filter. Mathematically, x(t) can be expressed in terms of its samples by:
x (t ) = x (kTs ) g (t kTs )
k
41
Natural Sampling
A delta function can be approximated by a
rectangular pulse p(t)
T1 , T2 t T2
p (t ) =
0, elsewhere.
h p (t ) =
p(t kT )
k =
It can be shown
that in
this case as well
the original
signal can be
reconstructed
from its samples
at or above
the Nyquist rate
through simple
low-pass filtering
42
Zero-Order-Hold Sampling
x s ( t ) = p ( t ) [x ( t ) h ( t ) ]
x(t)
P(f)
0
Ts
x s (t )
1
P( f )
Equalizer
G( f )
Low-pass
Filter
x (t )
Reconstruction is
possible but an
equalizer may be
needed
43
Practical Considerations of Sampling

Since in practice low-pass filters are not ideal and have a finitely steep roll-off, in
practice the sampling frequency fs is about 20% higher than the Nyquist rate:
f s 2.2W
50
0
-50
-100
-150
0
0.2
0.4
0.6
0.8
Example:
Music in general has a spectrum with frequency components in the range ~20kHz. The
ideal, smallest sampling frequency fs is then 40 Ksamples/sec. The smallest practical
sampling frequency is 44Ksamples/sec. In compact disc players, the sampling frequency
is 44.1Ksamples/sec.
44
Summary of Sampling (Nyquist) Theorem

An
analog signal of bandwidth W Hz can

be reconstructed exactly from its
samples taken at a rate at or above 2W
samples/s (known as the Nyquist rate)
x(t)
xs(t)
Ts
1
f =
s T > 2W
s
t
45
Summary of Sampling Theorem (contd)

Signal reconstruction
x(t)
xs(t)
Low-Pass
Filter
0
Amplitude still takes values on a continuum => Infinite number of bits

Need to have a finite number of possible amplitudes => Quantization
46
Quantization
Quantization
is the process discretizing the

amplitude axis. It involves mapping an infinite
number of possible amplitudes to a finite set
of values.
N bits can represent L = 2

amplitudes.
This corresponds to N-bit quantization
N
Quantization:
Uniform Vs. Nonuniform
Scalar Vs. Vector
47
Example (Quantization)
Let N=3 bits. This corresponds to L=8
quantization levels:
x(t)
111
110
101
100
000
3-bit
Uniform
Quantization
001
010
011
48
Quantization (contd)
There
is an irrecoverable error due to

quantization. It can be made small through
appropriate design.
Examples:
Telephone speech signals: 8-bit quantization
CD digital audio: 16-bit quantization
49
Input-Output Characteristic
7
2
3-Bit (8-level)
Uniform
Quantizer
Output,
x = Q ( x )
5
2
3
2
3
2
5
2
Input, x
Quantization Error:
d = ( x x ) = ( x Q ( x ) )
7
2
50
Signal-to-Quantization Noise Ratio (SQNR)

For Stochastic Signals
For Random Variables
PX
SQNR =
D
where :
1
PX = lim
T T
1
D = lim
T T
PX
SQNR =
D
where :
T
2
[ ]
PX = E X 2
T E X (t ) dt
2
T E [X (t ) Q ( X (t ))] dt
T
2
D = E ( X Q( X ))
Can be used for

stationary processes
51
SQNR for Uniform Scalar Quantizers

Let the input x(t) be a sinusoid of amplitude V volts. It can
be argued that all amplitudes in [-V,V] are equally likely.
Then, if the step size is , the quantization

error is
uniformly distributed in the interval

2 , 2
2
1 2 2
D = e de =
2
12
For an N-bit quantizer:
1/
/2
/2 e
V2
1 T2 2 2
PX = lim T V sin (t )dt =
T T
2
2
= 2V / 2 N
P
SQNR = 10 log10 X
D
p(e)
= 6.02 N + 1.76 dB
52
Example
A zero-mean, stationary Gaussian source X(t) having spectral density as given below
is to be quantized using a 2-bit quantization. The quantization intervals and levels are
as indicated below. Find the resulting SQNR.
SX ( f ) =
200
(
)
=
100
e
XX
2
1 + (2f )
PX = RXX (0 ) = 100
f X ( x) =
-5
-15
-10
D = E ( X Q( X )) = 2
2
10
( x 5)
5
0
1
e
200
x2
200
15
10
f X ( x )dx + 2 ( x 15) f X ( x )dx = 11.885

2
10
100
SQNR = 10 log10
= 9.25 dB
11.885
53
Non-uniform Quantization
In general, the optimum quantizer is non-uniform
Optimality conditions (Lloyd-Max):
The boundaries of the quantization intervals are the mid-points of the

corresponding quantized values
The quantized values are the centroids of the quantization regions.
Optimum quantizers are designed iteratively using the above rules
We can also talk about optimal uniform quantizers. These

have equal-length quantization intervals (except possibly the
two at the boundaries), and the quantized values are at the
centroids of the quantization intervals.
54
Optimal Quantizers for a Gaussian Source
55
Companding (compressing-expanding)
Uniform
Quantizer
Compressor
Low-pass
Filter
- Law Companding :
Expander
= 255
0.8
ln (1 + x )
g ( x) =
sgn( x ),
ln (1 + )
= 10
0.6
1 x 1
0.4
=0
0.2
0 0
1
g ( x) =
[
(1 + ) 1] sgn( x ),
0.2
0.4
0.6
0.8
0.8
1 x 1
=0
0.6
0.4
= 10
0.2
= 255
0
0.2
0.4
0.6
0.8
56
Examples: Sampling, Quantization

Speech
signals have a bandwidth of about

3.4KHz. The sampling rate in telephone
channels is 8KHz. With an 8-bit quantization,
this results in a bit-rate of 64,000 bits/s to
represent speech.
In CDs, the sampling rate is 44.1KHz. With a
16-bit quantization, the bit-rate to represent
(for each channel) is 705,600 bits/s (without
coding).
57
Data Compression
A/D Converter
Analog
Source
Sampler
Quantizer
Source
Encoder
001011001...
Discrete Source
Discrete
Source
01101001...
Source
Encoder
10011...
The job of the source encoder is to efficiently

(using the smallest number of bits)
represent the digitized source
58
Discrete Memoryless Sources

Definition: A discrete source is memoryless if successive
symbols produced by it are independent.
For a memoryless source, the probability of a

sequence of symbols being produced equals the
product of the probabilities of the individual
symbols.
59
Measuring Information
Not
all sources are created equal:
Example:
Discrete
Source 1
Discrete
Source 2
Discrete
Source 3
P(0)=1
No information provided
P(1)=0
P(0)=0.99
Little information is provided

P(1)=0.01
P(0)=0.5
Much information is provided

P(1)=0.5
60
Measuring Information (contd)

The amount of information provided is a function of
the probabilities of occurrence of the symbols.
Definition: The self-information of a symbol x which
has probability of occurrence p is
I(x)=-log2(p) bits
Definition: The average amount of information in

bits/symbol provided by a binary source with P(0)=p
is
H ( x ) = p log 2 ( p ) (1 p ) log 2 (1 p )
H(x) is known as the entropy of the binary source

61
The Binary Entropy Function

Maximum information is conveyed when the
probabilities of the two symbols are equal
H(x)
1
0.5
0.5
p
62
Non-Binary Sources
In general, the entropy of a source that produces L symbols with
probabilities p1, p2, ,pL, is
L
H ( X ) = pi log 2 ( pi ) bits
i =1
Property: The entropy function satisfies
0 H ( X ) log 2 (L )
Equality iff the source
probabilities are equal
63
Encoding of Discrete Sources

Fixed-length coding: Assigns source symbols binary
sequences of the same length.
Variable-length coding: Assigns source symbols
binary sequences of different lengths.
Example: Variable-length code
Symbol Probability
Codeword
Length
3/8
3/8
11
1/8
100
1/8
101
3
3
1
1
M = mi pi = 1 + 2 + 3 + 3
8
8
8
8
i =1
= 1.875 bits/symbol
4
H ( X ) = pi log 2 ( pi ) = 1.811 bits/symbol

i =1
64
Theorem (Source Coding)

The
smallest possible average number of

bits/symbol needed to exactly represent
a source equals the entropy of that
source.
Example 1: A binary file of length 1,000,000
bits contains 100,000 1s. This file can be
compressed by more than a factor of 2:
H ( x ) = 0.9 log(. 9 ) 0.1 log(. 1) = 0.47 bits
S = 106 H ( x ) = 4.7 105 bits

Compression Ratio=2.13
65
Some Data Compression Algorithms

Huffman
coding
Run-length coding
Lempel-Ziv
There are also lossy compression algorithms
that do not exactly represent the source, but
do a good job. These provide much better
compression ratios (more than a factor of
10, depending on reproduction quality).
66
Huffman Coding (by example)

A binary source produces bits with P(0)=0.1. Design a Huffman code that
encodes 3-bit sequences from the source.
Code bits
Source bits Probability
largest
Lengths
111
011
110
010
101
.081
001
011
.081
00011
100
.009
00010
010
.009
00001
001
00000
000
smallest
.729
.081
.009
.001
.162
1.0
0
.271
1
1
.109
.018 1
0
1
0
0
.028
0
.01
0
0
H ( x ) = [ 0.1 log2 (0.1) 0.9 log2 (0.9)] = 0.469

1
M = (1 0.729 + 3 3 0.081 + 5 3 0.009 + 5 0.001) = 0.53
3
67
Run-length Coding (by example)

A binary source produces binary digits with P(0)=0.9. Design a run-length code
to compress the source.
Source Bits
Run Lengths
Probability
Codewords
0.100
1000
01
0.090
1001
001
0.081
1010
0001
0.073
1011
00001
0.066
1100
000001
0.059
1101
0000001
0.053
1110
00000001
0.048
1111
00000000
0.430
M 1 = 4 (1 0.43) + 1 0.43 = 2.71

M 2 = .1 + 2 .09 + 3 .081 + 4 .073
+ 5 .066 + 6 .059 + 7 .053
+ 8 .048 + 8 .430
= 5.710
M =
M 1 2.710
=
= 0.475
M 2 5.710
H ( X ) = 0.9 log2 (0.9) 0.1 log2 (0.1)

= 0.469 < 0.475
68
Examples: Speech Compression

Toll-quality
speech signals can be produced at

8Kbps (a factor of 8 compression compared to
uncompressed telephone signals).
Algorithms
that produce speech at 4.8Kbps or

even 2.4Kbps are available, but have reduced
quality and require complex processing.
69
Example: Video Compression

Uncompressed
video of a 640x480 pixel2

image at 8 bits/pixel and 30 frames/s requires
a data-rate of 72 Mbps.
Video-conference
MPEG2
systems operate at 384Kbps.
(standard) operates at 3Mbps.
70

Learning Material - An Introduction To Digital Communication by Texas A&m University

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Learning Material - An Introduction To Digital Communication by Texas A&m University

Uploaded by

Copyright:

Available Formats

An Introduction to

Probability space, random variables, density

stationarity, spectral density

Sampling (ideal, natural, sample-and-hold)

coding (data compression)

Measuring information, entropy, the source coding

General binary and

Block codes, hard and soft-decision decoding,

through bandlimited channels

ISI, Nyquist pulses, sequence estimation, partial

through fading channels

Rayleigh fading, optimum receiver, performance

Digital vs. Analog Communication

Example: Analog Amplitude Modulation (AM)

Analog vs. Digital (contd)

Digital systems transmit signals from a

Binary digital communication systems

Data Rate=1/T bits/s

Digital systems are resistant to noise...

Optimum (Correlation) Receiver:

Advantages of Digital Systems

Easier data multiplexing

A General Digital Communication System

Some Probability Theory

a set , i.e., = {Ai ; Ai } is called an algebra of sets if:

Example: Let = {0,1,2} .

Definition: Let be a -algebra of a space . A function P

is a valid probability measure.

Definition: Let be a space, be a -algebra of subsets of , and

Random Variables and Density Functions

of a non-negative function f X () such that

We will refer to f X () as the density function of random variable X.

Example Density Functions

c) The Laplacian density function:

P[{2} | "odd outcome"] = 0

Let A1, A2,..., AN be a partition of , i.e.,

Then, the probability of occurrence of event B can expressed as

Illustration, Law of Total Probability

Example, Conditional Probability

P00 = 1 P01 = 0.99

P11 = 1 P10 = 0.99

Pr( e) = Pr( 0) P01 + Pr(1) P10 =

Ai , i = 1, 2,..., M be a partition of and B an event in . Then

Statistical Independence of Events

In general, the expected value of some function g( X ) of a random variable X is given by

When g( X ) = X n for n = 0,1,2,L , the corresponding expectations are referred to as the

Independence of Random Variables

The Characteristic Function

Example: The characteristic function of a Gaussian random X variable having mean

Definition: The moment-generating function of a random variable X is defined by

Characterization of Stochastic Processes

Consider a stochastic process { X ( ); < < } . The random variable X(t), t ,

We have E [ X ( t )] = 0 (zero-mean process), and var[ X ( t ) ] = t .

Autocovariance and Autocorrelation

Definition: The autocorrelation function of a random process X(t) is defined by

Definition: A random process X is uncorrelated if for every pair (t1,t2)

Definition: A process X is mean-value stationary if its mean is not a function of time.

RXX (t1 , t2 ) = exp t1 t2 = exp[

Fact: For a zero mean process X,

Linear Filtering of Stochastic Signals

The spectral density at the output of a linear

White Gaussian Noise

Analog signal: Continuous time,