Andra Boiek

Finite Wordlength Linear-Phase FIR Filter Design Using Babai's

Algorithm
Andra Boiek

andraz.bozicek@fri.uni-lj.si

Faculty of Computer and Information Science

University of Ljubljana
Ljubljana, 1000, Slovenia

Abstract
Optimal finite linear-phase impulse response (FIR) filters are most often designed using the
Remez algorithm, which computes so-called infinite precision filter coefficients. In many practical
applications, it is necessary to represent these coefficients by a finite number of bits. The problem
of finite wordlength linear-phase filters is not as trivial as it would seem. The simple rounding of
coefficients computed by the Remez algorithm gives us a suboptimal filter. Optimal finite
wordlength linear-phase FIR filters are usually designed using integer linear programming, which
takes a lot of time to compute the coefficients. In this paper, we introduce a new approach to the
design of finite wordlength FIR filters using very fast Babai's algorithm. Babai's algorithm solves
the closest vector problem, and it uses the basis reduced by the LLL algorithm as an input. We
have used algorithms which solve the problem in the L2 norm and then added heuristics that
improve the results relative to the L norm. The design method with Babai's algorithm and
heuristics has been tested on filters with different sets of frequency-domain specifications.
Keywords: FIR filter design, finite wordlength coefficients, Babai's algorithm, LLL algorithm,
closest vector problem.

1. INTRODUCTION
The design of optimal finite wordlength finite impulse response (FIR) filters can be formulated as
the Chebyshev approximation problem [1]. It is viewed as a criterion that the weighted
approximation error between the desired and the actual frequency response is spread evenly
accross the passband and stopband of the filter. This criterion minimizes the maximum absolute
error. There exist very good approximation algorithms (including the well-known Remez
algorithm), which give the optimal polynomial coefficients in the L norm. Standard approximation
algorithms yield unbounded or so-called "infinite precision" coefficients. In many practical
situations, we want to use cheaper and faster fixed-point digital signal processors (DSPs).
There are many approaches which yield the finite wordlength linear-phase coefficients, but not all
are optimal. The most simple approach is to round coefficients calculated by the Remez algorithm
to a desired length. However, this results in poor frequency response of the filter and suboptimal
coefficients [2]. Kodek [2] proposes the use of the mixed integer linear programming (MILP)
technique in the design of finite wordlength linear-phase FIR filters to give optimal coefficients.
The slowness is the only disadvantage of this technique. An approach which significantly speeds
up the calculation of coefficients is represented in [3]. By knowing the lower bound of
approximation error, the number of subproblems can be reduced. This also reduces the amount
of calculation. Derivation of an improved lower bound that uses the LLL algorithm is given in [4].
As was mentioned earlier, we can formulate the problem of optimal finite wordlength linear-phase
FIR filter design as a polynomial approximation. The polynomial approximation has been solved
with approaches based on the lattice theory algorithms, i.e. the LLL algorithm and Babai's nearest
plane algorithm [5]. The LLL algorithm is a polynomial-time lattice reduction algorithm, named
after its three authors. The formal description of the algorithm is in [6], and the implementation of

Signal Processing: An International Journal (SPIJ), Volume (6): Issue (5) : 2012

146

Andra Boiek

the algorithm, including the pseudo-code, can be found in [7]. The aim of the LLL algorithm is to
reduce the lattice basis, so the new lattice is equally described with shorter and almost orthogonal
vectors. The LLL algorithm has been successfully used in many areas, including practical
applications such as cryptography, GPS navigation, and wireless communications. Babai's
nearest plane algorithm solves the closest vector problem [8] and it uses the LLL reduced basis
as the input.

2. OPTIMAL FINITE WORDLENGTH LINEAR-PHASE FIR FILTERS

The frequency response 1 of an optimal infinite precision linear-phase FIR digital filter of length
is equal to
(1)
(2)

where
is a real function and
are the coefficients of the filter depending on filter length
(odd or even) and filter symmetry (positive or negative). There are exactly four types of linearphase FIR filters. The upper limit in the sum is
for type 1 filters,
for type 3
filters, and

for type 2 and type 4 filters.

is defined as the filter length. We also define

the desired frequency response

(which is defined to be unity in the passband and
stopband of the filter) and a weighting function
(which allows us to choose the relative size
of the approximation error in the passband and stopband of the filter). For mathematical
convenience, a modified weighting function
and a modified desired frequency response are
defined as
(3)
(4)
The weighted approximation error is defined as
(5)
To determine the filter cofficients
solved

, the following minimax approximation problem has to be

(6)
The set consists of the passbands and stopbands of the desired filter, e.g.
.
To make the finite wordlength constraint, the filter coefficients
are -bit integers from the set
, where
.
The most significant bit of the coefficient represents the sign and the other
the magnitude.

(7)
bits represent

3. POLYNOMIAL APPROXIMATION AND FINITE WORDLENGTH LINEAR

FIR FILTER DESIGN USING BABAI'S ALGORITHM AND HEURISTICS
To represent an arbitrary non-trivial mathematical function on a computer, one usually uses its
approximation. The approximation problem can be defined as a search for a function
, which
belongs to a given class of functions and is as close as possible to a function
. Because there

Signal Processing: An International Journal (SPIJ), Volume (6): Issue (5) : 2012

147

Andra Boiek

exist efficient schemes of polynomial evaluation, the approximation function

usually belongs
to a class of polynomials.
The typical approximation problem is to search the polynomial
degree
which is
sufficiently close to
. The distance between functions is defined using the norm. The quality
of the approximation is measured with the norm of the remainder
Different norms have
different approximation functions. The most usual are the L 2 and L norms. If a continuous
function
on an interval
is assumed, then the approximation problem can be defined as
the L2 norm searching the

, which minimizes
(8)

the L norm searching the

, which minimizes
(9)

The approximation polynomial can be a trigonometric polynomial. Finite wordlength FIR filter
design can be formulated as the problem of approximation by sums of cosines.
According to (5) and (6) the aim of the polynomial approximation is that the desired
frequency response
and polynomial
are as close as possible. If
is represented
as the vector and
as the vector

and

(10)

then we wish that vectors and are as close as possible according to the L norm. Frequencies
represents equally discretized frequencies in the interval
. The transition band
is, unlike to the Remez design method, observed in the interval. If
is rewritten as vectors

then integer coefficients

(11)

that minimize
(12)

have to be found.
Problem (11) can be formulated as the closest vector problem in the L norm. Kannan's
algorithm solves this problem in L, but its complexity is super-exponential [5]. In practice it is
better to use Babai's algorithm, which solves the problem in the L2 norm, and then use some
heuristics to improve the results relative to the L norm.
Babai's algorithm returns the vector as the result. In [5], heuristics similar to that used in
this paper are described. The heuristics assume that the result in the L norm is close to the
result in the L2 norm. The heuristics search the neighborhood of the vector . Vectors which
represent polynomial
(and are LLL-reduced) are added and subtracted from the vector . In

Signal Processing: An International Journal (SPIJ), Volume (6): Issue (5) : 2012

148

Andra Boiek

this manner, we get candidates for a result which is, according to the L norm, better than the
previous result. The L norm is calculated between the candidates in the desired frequency
response, and if the L norm is lower than it was for the previous result than we keep the new
result. This procedure is repeated until the result is improved. The heuristics are described with
the pseudocode presented in Table 1.
Description: Explore the neighborhood of the vector
to get close to the desired
frequency response (vector ) according to the L norm
Input: vector , vector LLL-reduced basis
The vector represents the frequency response of the filter and is according to the L 2
norm as close as possible to the desired frequency response of the filter. The vector
represents the desired frequency response. The LLL-reduced basis
includes vectors
.
Output: vector .
Set is filled with
vectors:
.
Reference norm is calculated
.
For
to
calculate
.
If exists
such that
,
then new
and goto 2.
Else return .
TABLE 1: Heuristics pseudocode

The disadvantage of the finite wordlength linear-phase FIR filter design method described is that
the weighting function
cannot be observed in the design process.

4. RESULTS
Our design method has been tested on 25 filters with five different sets of frequency-domain
specifications. The frequency domain specifications are given in Table 2.
Set

Band 1

Band 2

pass

stop

pass

stop

pass

stop

Band 3

]
pass
]
D

pass

stop

pass

stop

pass

TABLE 2: Characteristics of filter test sets

Signal Processing: An International Journal (SPIJ), Volume (6): Issue (5) : 2012

149

Andra Boiek

Initially, filter A with 45 eight-bit coefficients was synthesized. As can be seen in Table 3, the
design method with Babai's algorithm and heuristics did not give us the optimal filter, but the
result was better than using rounding of coefficients calculated by the Remez algorithm. The
calculation time of Babai's algorithm and heuristics was very short compared to the MILP method.
The heuristics have also slightly improved the result of Babai's algorithm. Fig. 1 shows the
magnitude response for the above filter designed using the MILP and Babai's algorithm with
heuristics.

Max. deviation
Passband
deviation
Stopband
deviation
Calculation time

Rounding
0.037059
0.03125
(0.267279 dB)
0.037059
(28.622046 dB)
0.064 s

Babai's
algorithm
0.032884
0.029542
(0.252882 dB)
0.032884
(29.660432 dB)
0.192 s

Babai's
algorithm with
heuristics
0.032668
0.032668
(0.279217 dB)
0.031262
(30.099566 dB)
0.210 s

MILP
0.028847
0.028847
(0.247016 dB)
0.027691
(31.153227 dB)
44.431 s

TABLE 3: Results for filter A, 45 eight-bit coefficients

FIGURE 1: Magnitude response for filter from set A (length 45, 8-bit coefficients)

The non-unit weighting function could not be used in the design process. When designing filters
from sets B and D, the magnitude response from the Remez algorithm was approximated. In
these cases, the results were not very good. In all other cases, results were better than using the
simple rounding technique, and in some cases optimal filter coefficients were calculated. The
results can be seen in Table 4.

Signal Processing: An International Journal (SPIJ), Volume (6): Issue (5) : 2012

150

Andra Boiek

Set
A

M=25, b=7
M=35, b=8
M=45, b=8
M=45, b=10
M=55, b=10
M=25, b=8
M=35, b=9
M=45, b=9
M=45, b=11
M=55, b=11

M=25, b=7
M=35, b=8
M=45, b=8
M=45, b=10
M=55, b=10
M=25, b=8

M=35, b=9

M=45, b=9

M=45, b=11

M=55, b=11

M=25, b=7
M=35, b=8
M=45, b=8
M=45, b=10
M=55, b=10

max. deviation
max. deviation
max. deviation
max. deviation
max. deviation
passband deviation
stopband deviation
passband deviation
stopband deviation
passband deviation
stopband deviation
passband deviation
stopband deviation
passband deviation
stopband deviation
max. deviation
max. deviation
max. deviation
max. deviation
max. deviation
passband 1 deviation
stopband deviation
passband 2 deviation
passband 1 deviation
stopband deviation
passband 2 deviation
passband 1 deviation
stopband deviation
passband 2 deviation
passband 1 deviation
stopband deviation
passband 2 deviation
passband 1 deviation
stopband deviation
passband 2 deviation
max. deviation
max. deviation
max. deviation
max. deviation
max. deviation

Rounding
0.078125
0.032668
0.037059
0.013872
0.00979
0.140625
0.032914
0.074219
0.015902
0.03801
0.011719
0.024179
0.006177
0.010579
0.006234
0.041507
0.046875
0.030466
0.00993
0.010839
0.0625
0.014408
0.0625
0.015987
0.012202
0.027344
0.01768
0.010906
0.018901
0.004244
0.003222
0.003906
0.002167
0.00371
0.004379
0.077633
0.046934
0.035784
0.015263
0.011802

Babai's algorithm with heuristics

0.065237
0.032668
0.032668
0.010182
0.009144
0.143501
0.039063
0.066406
0.022931
0.034366
0.015908
0.025522
0.006282
0.011861
0.004939
0.036676
0.03125
0.029409
0.006843
0.007636
0.0625
0.014408
0.0625
0.023438
0.011957
0.023438
0.014144
0.011689
0.011097
0.003761
0.002954
0.004551
0.002167
0.00371
0.004379
0.062141
0.033004
0.035784
0.011185
0.009119

MILP
0.065237
0.029966
0.028847
0.010182
0.008296
0.154246
0.015234
0.073276
0.007313
0.052641
0.005681
0.026655
0.002673
0.01666
0.001643
0.036676
0.016767
0.016085
0.004761
0.004365
0.078125
0.007966
0.078125
0.03125
0.003302
0.03125
0.022836
0.002552
0.025071
0.006396
0.000745
0.005306
0.00518
0.000618
0.005859
0.06071
0.032888
0.028987
0.010104
0.008199

TABLE 4: Results

5. CONCLUSION
A new method using Babai's algorithm and heuristics for the design of finite wordlength linearphase FIR filters was presented in this paper. Heuristics were used to improve the Babai's
algorithm result and to bring the result closer to the L norm. Testing showed that the heuristics
improve the result of Babai's algorithm. The major advantage of this design method is the speed
of the algorithm. Major disadvantages are that we do not get always the optimal filter coefficients
and we cannot design filters with non-unit weighting functions. These will be the main focus of our
future research, and our aim is to minimize those disadvantages.

Signal Processing: An International Journal (SPIJ), Volume (6): Issue (5) : 2012

151

Andra Boiek

Our future research will also attempt to include some other concepts. The problem of
searching the closest point in a lattice is in communications community referred as the sphere
decoding. Using results of [9], [10] some of the improvements over the Babai's algorithm could be
reached. In [11], [12] the peak constrained least squares method is introduced. This method
balances the minimax and the squared error criteria. This approach could be useful framework to
pose the filter design problem.

6. REFERENCES
[1] J. G. Proakis and D. G. Manolakis, Introduction to Digital Signal Processing., pp. 614-726.
[2] D. M. Kodek, "Design of optimal finite wordlength FIR digital filters using integer
programming techniques," IEEE Transactions on Acoustics, Speech and Signal Processing,
vol. 38, no. 3, pp. 304- 308, junij 1980.
[3] D. M. Kodek, "Performance Limit of Finite Wordlength FIR Digital Filters," IEEE Transactions
on Signal Processing, vol. 53, no. 7, pp. 2462- 2469, julij 2005.
[4] D. M. Kodek, "LLL Algorithm and the Optimal Finite Wordlength FIR Design," IEEE
Transactions on Signal Processing, vol. 60, no. 3, pp. 1493-1498 , March 2012.
[5] N. Brisebarre and S. Chevillard, "Efficient polynomial L-approximations," Computer
Arithmetic, 2007. ARITH '07. 18th IEEE Symposium on, pp. 169-176, junij 2007.
[6] A. K. Lenstra, H. W. Lenstra, and L. Lovsz, "Factoring polynomials with rational
coefficients," Mathematische Annalen, vol. 264, no. 4, pp. 515-634, december 1982.
[7] N. P. Smart, The algorithmic resolution of diophantine equations.: Cambridge University
Press, 1998, pp. 59-76.
[8] L. Babai, "On Lovsz' Lattice Reduction and the Nearest Lattice Point Problem," in
Proceedings of the 2nd Symposium of Theoretical Aspects of Computer Science (STACS
'85), London, 1985, pp. 13-20.
[9] E. Agrell, T. Eriksson, A. Vardy, and K. Zeger, "Closest point search in lattices," IEEE
Transactions on Information Theory, vol. 48, no. 8, pp. 2201-2214, August 2002.
[10] A. D. Murugan, H. El Gamal, M. O. Damen, and G. Caire, "A unified framework for tree
search decoding: rediscovering the sequential decoder," IEEE Transactions on Information
Theory, vol. 52, no. 3, pp. 933-953 , March 2006.
[11] J. W. Adams and J. L. Sullivan, "Peak-constrained least-squares optimization," IEEE
Transactions on Signal Processing, vol. 46, no. 2, pp. 306-321, February 1998.
[12] I. W. Selesnick, M. Lang, and C. S. Burrus, "Constrained least square design of FIR filters
without specified transition bands," IEEE Transactions on Signal Processing, vol. 44, no. 8,
pp. 1879-1892, August 1996.

Signal Processing: An International Journal (SPIJ), Volume (6): Issue (5) : 2012

152