IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 48, NO.
3, MAY 1999
Sequence Detection and Channel State Estimation
over Finite State Markov Channels
Hongwei Kong and Ed Shwedyk
Abstract—A useful model for general time-varying channels is
a finite state Markov chain. In this paper, maximum likelihood
sequence estimation (MLSE) for signals over finite state Markov
channels (FSMC’s) is studied. Also studied is the maximum a
posteriori (MAP) channel state estimation. When coded signals
with interleaving are transmitted, the channel estimates can be
used to make soft-decision decoding. The error performance of
the proposed sequence and channel state estimation schemes are
evaluated through computer simulations. The effect of channel
modeling error is also discussed.
I. INTRODUCTION
T HE FINITE state Markov chain has been used to model
general time-varying channels with memory [1], [2].
The state space usually consists of a finite set of additive
white Gaussian noise (AWGN) channels, either bandlimited
or unlimited, with each state corresponding to a specific
channel quality indicated by the signal to noise ratio (SNR)
or bandwidth. The output from a finite state Markov channel
(FSMC) depends on the present input and the channel state
while the next channel state only depends on the present
channel state and is independent of the input. Since the
channel state is not directly observable at the receiver but
is embedded in the received signal, the FSMC model is also
referred to as a hidden Markov model (HMM), a terminology
that comes from speech recognition [3]. For the purpose of
channel modeling, a -state FSMC model is specified by the
state space and the transition
probabilities between states Pr
for , where is the channel state at
time . A stationary FSMC then has transition probabilities
that are independent of the time index . with interleaving, even when per-survivor processing is used,
The simplest FSMC model is the two-state Markov channel
model studied by Gilbert [4] and Elliott [5]. To model time-
varying channels whose quality changes over a wide dynamic
range, an FSMC model with more states is required. In [2],
the FSMC model is applied to model the amplitude process
of frequency nonselective Rayleigh fading channels. In that
model, the received SNR is partitioned (quantized) into a finite
set of intervals with each interval corresponding to a state in
the FSMC. Then the FSMC model is constructed by finding
Manuscript received January 18, 1997; revised May 5, 1998. This work
was supported by a University of Manitoba research scholarship and by an
operating grant from NSERC (Canada).
H. Kong is with the Wireless Technologies Laboratories, Lucent Technolo-
gies, Whippany, NJ 07981 USA.
E. Shwedyk is with the Department of Electrical and Computer En-
gineering, University of Manitoba, Winnipeg R3T 2N2, Canada (e-mail:
shwedyk@ee.umanitoba.ca).
Publisher Item Identifier S 0018-9545(99)04037-2.
0018–9545/99$10.00  1999 IEEE
833
the transition probabilities using level crossing rates. An alter-
native construct of the FSMC model is given in the Appendix.
The FSMC model characterizes the correlated variation of
the time-varying channel. This Markovian structure should be
used in sequence detection. A typical time-varying channel is
the Rayleigh fading channel. Maximum likelihood sequence
estimation (MLSE) for signals transmitted over Rayleigh
fading channels has been studied in [6] using a vector
Gauss–Markov model, where the fading channel is modeled
as a continuous amplitude Markov process. For FSMC’s,
little work has been done in terms of sequence detection
and channel estimation. In [1], a two-stage decision feedback
decoder was proposed for the detection of an interleaved
coded signal over an FSMC. The first stage estimated the
channel state using the decision feedback from the second
stage which performed maximum likelihood decoding. This
is one variant of the so-called “reduced-state” MLSE [7].
Obviously, this receiver suffers from error propagation. It
was shown that when error propagation was ignored the
decoder preserved the channel capacity.
In this paper, the problem of maximum likelihood sequence
estimation for signals transmitted over an FSMC in additive
white Gaussian noise is studied. The receiver structure is
derived. However, there is no trellis structure for the receiver.
One possibility is to use per-survivor processing [7] which,
as the number of survivors increases, will eventually result
in optimum performance. This is done in the simulation. The
receiver eliminates the error propagation associated with the
decision feedback receiver of [1].
Though the derived receiver can be applied to coded signals
the receiver complexity is still prohibitive. Therefore, when a
coded signal with interleaving is transmitted over an FSMC,
the two-stage receiver of [1] is adopted. The channel state
sequence is first estimated using a maximum a posteriori
(MAP) criterion and the Viterbi algorithm to find the optimum
channel state sequence. The channel estimates are used in the
second stage to make soft-decision decoding [1], [8].
The rest of the paper is organized as follows. In Section II,
the MLSE receiver is derived. Section III discusses the MAP
channel state estimation. Both receivers have a metric that
involves a sum of exponentials. The joint channel state and
sequence estimation described in Section IV eliminates this.
The error performance of the sequence detection and chan-
nel state estimation schemes are evaluated through computer
simulations and the results are presented in Section V. In
Section VI, the problem of channel modeling error is discussed
and conclusions are drawn in Section VII.
834 IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 48, NO. 3, MAY 1999

II. MLSE OVER FSMC’s where is the

The channel is assumed to distort the transmitted signal probability of the th channel state at time conditioned on
through a multiplicative stationary random process (frequency the previous received signal samples and the specific sequence
nonselective). The received signal can be written as considered. The reason for is that is independent of
future data symbols. results from the fact that given the
(1) present channel state and the present input , the output
is statistically independent of and ,
where is the th-transmitted information symbol, is and similarly the present channel state is statistically
a sample of additive white Gaussian noise, and is the independent of the present input given and
length of the transmitted sequence. Note that the multiplicative .
term represents the FSMC and takes values from a finite size A recursive formula exists to calculate [1]
alphabet . is assumed to be a
stationary Markov process with transition probabilities given
by

(2) (5)

For notational convenience, define

and . They represent, respec-
tively, sequences of the received signal and the th possible
transmitted signal from the beginning up to time . Note
that and is the set of all
possible transmitted sequences for the -ary QAM signals.
For uncoded sequences where each symbol is chosen from
the modulation alphabet independently, and, equally likely,
the maximum a posteriori sequence estimate is equivalent to
the maximum likelihood sequence estimate. This maximum
likelihood (ML) sequence estimate is given by
(3)
where argmax means maximizing the likelihood function over
all the possible transmitted data sequences.
Usually, , , and the additive white Gauss-
ian noise are statistically independent. Under this condition,
one has
for . The two conditional probability
density functions in (5) are Gaussian density functions.
Although taking the logarithm of (4) on both sides gives an
additive metric, the Viterbi algorithm cannot be readily applied
here to find the ML sequence because from (5) it is evident that
depends on the entire history of the received signal. In
essence, even though , , , and are finite order Markov
sequences, is not in general. Thus, the received sequence
cannot be represented by a trellis, but only by a tree graph.
In this case, tree search algorithms such as the sequential
algorithm [6] can be used to find the optimum sequence. On
the other hand, when can be approximately considered
to be a finite order Markov process, the Viterbi algorithm can
be applied to find the (sub)optimum sequence by searching
through a trellis. In this paper, the latter is considered. If
is assumed to be of th order then the trellis will have
states. As the value of increases, the performance using a
trellis search approaches that of the optimum receiver.

III. MAP CHANNEL STATE ESTIMATION

Interleaving is a powerful technique to combat the memory
due to fading. Thus, coding with interleaving is considered
when the channel is an FSMC. Joint ML detection/decoding
based on per-survivor processing is now even more compli-
cated because of interleaving. Therefore, a two-stage subopti-
mal receiver is used where the first stage estimates the channel
state and the second stage uses the channel estimate from the
first stage to perform soft-decision decoding [7], [8]. Unlike
channel estimation for the Rayleigh fading channel where
an optimum estimate is impossible to obtain [16], an FSMC
model enables one to do an optimum channel state estimation.
The optimum (maximum a posteriori probability) channel
state estimate is given by

(6)

(4) where the maximization is over all the possible channel state
sequences.
KONG AND SHWEDYK: SEQUENCE DETECTION AND CHANNEL STATE ESTIMATION 835

Since

(7)

and is the same for all possible , the equivalent

condition for (6) is
(8) Fig. 1. Transition diagram for the two-state Markov model.

But
is a constant if the transmitted sequences
are equally probable. Therefore, the joint MAP estimate in
(11) can be obtained from

(13)
where

(14)
(9)
Taking the logarithm of (14) gives an additive metric. The
The first step in (9) comes as a result of the Markov as- branch metric depends on . Joint estimation
sumption. Assume successive symbols are equally probable can be done by searching through a trellis with states,
and statistically independent, i.e., for -ary where is the size of the modulation alphabet (accounting for
modulation. Thus, (9) becomes ) and is the number of states for the FSMC (accounting
for ) [9].

(10)
An additive metric can be obtained by taking the logarithm of
(10) and the Viterbi algorithm can be applied to search for the
optimum channel state sequence.
IV. JOINT SEQUENCE DETECTION AND CHANNEL ESTIMATION
The metric derived in the previous sections [(4) and (10)]
involves the calculation of a sum of exponential terms over the
alphabet of either the channel or the signal constellation. In
this section, a receiver that does both sequence detection and
channel state estimation simultaneously is derived. It avoids
the computation of the exponential terms in (4) and (9).
The joint MAP channel and sequence estimate is given by
(11)
Under the assumption that and are statistically
independent, one has
(12)
V. SIMULATION RESULTS
The performance of the above receivers is studied and com-
pared through computer simulations. The 4-ASK modulation
is considered in all the simulations in this section. ASK is
chosen because the FSMC used in this section is based on the
amplitude characteristics of the fading channel [1], [2]. Phase
distortion is not included. The first FSMC model considered is
the two-state model [1] with no signal attenuation for the good
state and 20-dB attenuation for the bad state. Thus, is
a two-state Markov model with the transitional diagram given
in Fig. 1.
Both a reduced-state MLSE receiver with one state and
the joint MAP channel and sequence estimation receiver
were simulated. The one state MLSE receiver is the decision
feedback receiver discussed in [1]. Fig. 2 shows the probability
of bit error versus the average received SNR for a channel
model with transition probability . The
error performance with ideal channel state information is also
included for reference. One sees that even for nonconstant
amplitude modulation, MLSE and joint MAP are equivalent
in terms of sequence estimation. Both sequence detection
schemes achieve the performance of coherent detection (ideal
CSI).
The performance of the channel estimation schemes were
also determined for the above two-state FSMC model. The
results are given in Fig. 3. Since the channel has a finite
number of states, the figure of merit here is the probability
of channel state estimation error. As expected, the channel
estimate from the joint MAP channel and sequence estimation
receiver converges to the MAP estimate of the channel, i.e., the
836
Fig. 2. Error performance of MLSE and joint MAP estimation for a two-state
FSMC.
Fig. 3. Channel estimation error of MAP and joint MAP for a two-state
FSMC.
channel estimate from the joint MAP is asymptotically MAP.
This convergence is achieved even though the modulation is
nonconstant amplitude. As a lower bound for both channel esti-
mation schemes, the channel estimate with a known sequence
is also shown.
For radio channels where the channel quality can change
dramatically, a two-state FSMC is not adequate [2]. As shown
in [2], to model fading channels with Rayleigh amplitude, at
least an eight-state FSMC model is required. Using the method
given in the Appendix, the FSMC model is derived for two
Rayleigh distributed random processes (land mobile channel),
one for a fast fading channel with and the other a
slow fading channel with . In Figs. 4 and 5, the
error performance for the two FSMC models is given for the
reduced-state MLSE receivers, as well as for the joint MAP
receiver. Phase distortion is assumed to be fully compensated
and is not considered in the simulation.
The results indicate that the error performance for the MLSE
receiver improves as the number of states in the trellis increase
from one to four. Increasing the number of states beyond
four does not improve the error performance very much. The
results also show that though the modulation is nonconstant
IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 48, NO. 3, MAY 1999
Fig. 4. Error performance of MLSE and joint MAP for a fast fading
eight-state FSMC.
Fig. 5. Error performance of MLSE and joint MAP for a slow fading
eight-state FSMC.
amplitude, the joint MAP sequence estimation receiver and the
MLSE receiver have very close bit error performance.
Figs. 6 and 7 show the performance of different chan-
nel estimation schemes for the above two FSMC models.
For both channels, the joint MAP channel estimation gives
asymptotically the MAP channel estimate and for slower
fading channels, the channel estimate approaches the lower
bound. Once again, this is achieved for nonconstant amplitude
modulation.
VI. DISCUSSION
The FSMC characterizes the memory of the channel. The
receiver structures derived above for both sequence detection
and channel estimation are generic, i.e., as long as an FSMC
model exists for the radio channel, the receiver can be applied.
This is quite attractive for radio channels with distributions
other than complex Gaussian. In this situation, it is very
difficult, if not impossible, to find the joint probability density
function for the received signal, hence the design of MLSE
or symbol-by-symbol MAP detection is not possible. The
FSMC characterization of radio channels provides a different
approach to the signal detection problem for this situation [10].
KONG AND SHWEDYK: SEQUENCE DETECTION AND CHANNEL STATE ESTIMATION
Fig. 6. Channel estimation error of MAP and joint MAP for a fast fading
eight-state FSMC.
Fig. 7. Channel estimation error of MAP and joint MAP for a slow fading
eight-state FSMC.
One thing that should be realized about the FSMC model is
that it is only an approximation to actual radio communication
channels. The impulse response of a radio channel is always
continuous. Thus, an FSMC model with a finite number
of states represents the channel analogously to the way a
quantized signal represents its analog counterpart. Obviously,
an FSMC with more states results in a better representation.
From (4) and (5), this means a linear increase in the receiver
complexity.
Another issue is that the FSMC has a memory of one
symbol, whereas practical radio channels have a memory that
could be several symbols long. For example, it is shown
using an information-theoretic measure [11] that for Rayleigh
fading channels with a Jakes’ correlation function [12], the
fading process has a memory length of about five. But the
amplitude (Rayleigh) process of the complex Gaussian fading
process has been shown [13] to be approximately first-order
Markovian. The design of MLSE receivers for higher order
FSMC’s is straightforward but the receiver complexity grows
exponentially with this order.
The modeling error (finite state and finite memory) using an
FSMC results in a degradation in error performance if a MLSE
837
Fig. 8. A comparison of the optimum receiver and the ML receiver based
on FSMC for the Rayleigh fading channel.
receiver based on the FSMC is to be used for the actual radio
channel. The degradation of error performance due to the use
of an FSMC model in receiver design can be most readily seen
for Rayleigh fading channels, since an optimum receiver is
available. Fig. 8 gives the results from computer simulations
for a Rayleigh fading channel with the correlation function
given below
(15)
where is the maximum Doppler frequency and is the
0th order Bessel function of the first kind. The normalized
maximum Doppler frequencies of (fast fading)
and 0.01 (slow fading) are used here. QPSK is chosen as the
modulation scheme, with a pilot insertion rate of 2 : 1 to resolve
the phase ambiguity.
A reduced-state MLSE with 16 states is used here as the
optimum receiver for the Rayleigh fading channel. For the
receiver based on the FSMC model, the method given in
the Appendix is used to obtain individual eight-state FSMC
models for the in-phase and the quadrature components of the
complex Gaussian process. Thus the complex fading process
is represented by a 64-state FSMC model. This construction is
quite straightforward and by no means optimum for the two-
dimensional (2-D) random process. Procedures described in
[15] can be used to reduce the number of states while retaining
the same error performance. An alternative is to construct the
Markov model based on a polar representation of the fading
process where the amplitude is Rayleigh distributed and the
phase is uniform. An obvious advantage of this approach is
that it could deal with the phase jump problem due to Rayleigh
fading.
The results show that the performance degradation due to
the FSMC modeling is relatively small for low-to-medium
SNR (within 1 dB when the SNR is from 5 to 15 dB).
The difference is increased at higher SNR. At these SNR’s
the channel modeling error becomes the dominant factor that
influences the error performance.
VII. CONCLUSIONS
Maximum likelihood sequence estimation for the FSMC is
studied in this paper. Compared with the decision feedback
838 IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 48, NO. 3, MAY 1999

receiver proposed in [1], error propagation has been elimi-

nated. Although a trellis is not available, computer simulation
indicates that per-survivor processing gives near optimum
performance when a relatively small number of states are
considered (e.g., four states for a 4-AM modulation).
The maximum a posteriori channel state estimation is also
considered. Unlike the channel estimation for Rayleigh fading
channels, the MAP channel state estimation for FSMC can be
done using the Viterbi algorithm. The channel state estimates
can be used in a two-stage receiver [1] for soft-decision
decoding. Fig. 9. Optimum quantization of a Rayleigh distributed random variable.
Both the MLSE and the MAP channel state estimation
receivers have a metric that involves the computation of a sum
The threshold for each interval can be found by solving
of exponentials. The joint MAP sequence and channel state
estimation approach avoids this problem, but still provides an for (A.2)
error performance that is close to the previous receivers.
The FSMC is an approximation to the actual time-varying The mean value of each small interval is given by
channel because of its finite number of states. The effect
of channel modeling error on the error performance of the (A.3)
receiver is also discussed. For Rayleigh fading channels, it is
found that for medium SNR’s, receivers based on FSMC give and is used as the state for the FSMC model.
close to optimum error performance. When is Rayleigh distributed with
Generalization of this receiver to frequency selective chan-
nels is quite straightforward [14]. (A.4)

where represents the unit step function, one has

APPENDIX
CONSTRUCTION OF A FINITE STATE MARKOV
MODEL FROM A GIVEN RANDOM PROCESS (A.5)
In this Appendix, a method for constructing an FSMC
Generally, if a closed-form expression for the left-hand side
model from a given random process is discussed. The channel
of (A.2) does not exist, numerical methods have to be used
considered is frequency nonselective, i.e., the received signal
to solve for the .
samples are given by
After the are obtained, the transition probability from one
state (quantization interval) to another can be calculated by
(A.1)

The construction of the FSMC model is done in two steps.

The first step constructs a state space by partitioning the
received SNR into a finite number of nonoverlapping regions.
For the frequency nonselective channel given in (A.1), this is
(A.6)
equivalent to partitioning into a finite number of regions.
In other words, the state space consists of the quantization
intervals of . After the state space is set up, transition When is the square root of the squared sum of two
probabilities between states are found in the second step. In independent identically distributed (i.i.d.) Gaussian random
[2], based on the level crossing rates the transition probabilities processes, it is Rayleigh distributed with the following 2-D
are derived for Rayleigh distributed . In this Appendix, a probability density function
different method is proposed to construct the FSMC model.
It is based on the one-dimensional (1-D) and 2-D joint
probability density functions of the fading process.
Let and be the 1-D and 2-D probability
density functions of the random process . (A.7)
To construct the state space, an optimal quantization method
that minimizes the mean square quantization error can be used where , is the anticorrelation function
to partition into nonoverlapping intervals. Fig. 9 shows of the underlying Gaussian process. is the zero-order
the quantization of a Rayleigh distributed . The range of modified Bessel function of the first kind.
is partitioned into small nonoverlapping intervals where Thus, to construct the FSMC model the 1-D and 2-D
each small shaded area in the graph has equal area . probability density functions need to be known analytically.
KONG AND SHWEDYK: SEQUENCE DETECTION AND CHANNEL STATE ESTIMATION
In situations where they are unknown, Monte Carlo simulation
has to be used to construct an FSMC model from experimental
data.
ACKNOWLEDGMENT
The authors wish to thank the reviewer who suggested the
advantages of 2-D quantization.
REFERENCES
[1] A. J. Goldsmith and P. Varaiya, “A decision-feedback maximum-
likelihood decoder for finite-state Markov channels,” in Proc. 1994 IEEE
Int. Conf. Commun., New Orleans, LA, 1994, pp. 1546–1550.
[2] H. S. Wang and N. Moayeri, “Finite state Markov channel—A useful
model for radio communication channels,” IEEE Trans. Veh. Technol.,
vol. 44, pp. 163–171, Feb. 1995.
[3] L. R. Rabiner and B. H. Juang, “An introduction to hidden Markov
models,” IEEE ASSP Mag., vol. 3, pp. 4–16, Jan. 1986.
[4] E. N. Gilbert, “Capacity of a burst-noise channel,” Bell Syst. Tech. J.,
vol. 39, pp. 1253–1265, Sept. 1960.
[5] E. O. Elliott, “Estimates of error rates for codes on burst-noise channels,”
Bell Syst. Tech. J., vol. 42, pp. 1977–1997, Sept. 1963.
[6] Q. Dai and E. Shwedyk, “Detection of bandlimited signals over fre-
quency selective Rayleigh fading channels,” IEEE Trans. Commun., vol.
42, pp. 941–950, Feb./Mar./Apr. 1994.
[7] G. M. Vitetta and D. P. Taylor, “Maximum likelihood decoding of
uncoded and coded PSK signal sequences transmitted over Rayleigh
flat-fading channels,” IEEE Trans. Commun., vol. 43, pp. 2750–2758,
Nov. 1995.
[8] H. Kong and E. Shwedyk, “On channel estimation and sequence
detection of coded signals over frequency nonselective Rayleigh fading
channels,” IEEE Trans. Veh. Technol., vol. 47, pp. 558–565, May 1998.
[9] , “Sequence estimation for frequency nonselective channels using
a hidden Markov model,” in Proc. IEEE 44th Veh. Technol. Conf.,
Stockholm, Sweden, June 1994, pp. 1251–1253.
[10] , “A hidden Markov model (HMM)-based MAP receiver for
Nakagami fading channels,” in Proc. 1995 IEEE Int. Symp. IT (ISIT),
Whistler, B.C., Canada, Sept. 1995, p. 210.
839
[11] , “A measure for the length of probabilistic dependence for
Gaussian and lognormal distributed random processes,” in Proc. Int.
Symp. Information Theory, Ulm, Germany, June 29–July 4, 1997, p.
469.
[12] W. C. Jakes, Ed., Microwave Mobile Communications. New York:
IEEE Press, 1974.
[13] H. S. Wang, “On verifying the first-order Markovian assumption for a
Rayleigh fading channel model,” in Proc. 3rd Annu. Int. Conf. Universal
Personal Communications, San Diego, CA, Sept. 1994, pp. 160–164.
[14] H. Kong and E. Shwedyk, “Markov characterization of frequency
selective Rayleigh fading channels,” in Proc. IEEE Pacific Rim Conf.
Commun., Computers and Signal Processing, Victoria, B.C., Canada,
May 1995, pp. 359–362.
[15] A. Gersho and R. M. Gray, Vector Quantization and Signal Compression.
Boston, MA: Kluwer, 1992.
Hongwei Kong received the B.S. and M.S. degrees from Beijing Broadcasting
Institute, China, in 1985 and 1988, respectively, and the Ph.D. degree from
the University of Manitoba, Winnipeg, Canada, in 1997.
From 1988 to 1993, he was with the Department of Television Engineering,
Beijing Broadcasting Institute, engaged in research and development of
enhanced broadcasting systems. During this time, he also taught some
courses and served as a consultant to several companies in the video and
broadcasting industry. From 1997 to 1998, he worked on cable modems
at 3Com/US Robotics. Presently, he is a Member of the Technical Staff,
Lucent Technologies, Wireless Technologies Laboratories, Whippany, NJ. His
research interests are in digital signal processing for wireless communications.
Ed Shwedyk received the B.Sc.(E.E.) and M.Sc.(E.E.) degrees from the
University of Manitoba, Winnipeg, Canada, in 1965 and 1968, respectively,
and the Ph.D. degree from the University of New Brunswick, Canada, in 1974.
Since 1974, he has been with the Department of Electrical and Computer
Engineering, University of Manitoba. His research interests are in digital com-
munications, principally estimation and detection, and in biosignal processing.