I
TRANSACTIONS O N SYSTTMS, MAN, ANI1 C.YHrRNrTIC.S, VOI . 20. NO. 3.
B. Peden and K. Sullivan, “Bounded deviation trajectory interpolation
for robot manipulators.“ presented I E E E Int. Conf. on Robotics and
Automation, 19x8.
H. Spath, Splirie A/gor-ir/ini,s for Cirrr.c.\ und . S i n f i i w \ . Utilitas Mathc-
matics, 19x4.
J. A. Adams, “Cubic spline curve fitting with controlled end conditions,“
Cornpiit. Aided Iksign, vol. 6, no. I , pp. 2-9. J a n . 1974.
I. D. Faux and M. J. Pratt, Coinpiirurronul (;cwrn<,/r.\.for- Dcsign arid
Muni!fucrior. Ellis Honvood, 19x1.
R. Featherstone, “Position and velocity transformations between robot
end effector coordinates and joint angles.” lnr. J. Kohor. Res.. vol. 2.
no. 2. 19x3.
S. Elgazzar, “Efficient kinematic transformations for the PUMA 500
robot.” l E E E J. Roh(~r.Airtornur.. vol. RA-I. no. 3, pp. 142-151. Sept.
IYXS.
D. E. Whitney, “The mathematics o f coordinated control of pro\thetic
arms and manipulators,” Truns. ASME J. D y n . Syv. Mvusirr-.Conrr.. vol.
94, no. 4, pp. 303-309, 1972.
J. M. Brady et U / . , Robot Morion: P l u m i n g cind Coiirrol. Cambridge:
MIT Press, 1983.
R. P. Paul and C. N . Stevenson, “Kincmatics of rohotics wrists,“ /N. J.
Rohor. Res.,vol. 2, no. I. pp, 31-3X. 19x3.
Z. C. Lai and D. C. H. Yang, “ A new method for the singularity
analysis of simple six-link manipulators,” / ? I / . J. Rohor. Kc\.,vol. 5 , no.
2, 19x6.
Information Fusion in Computer Vision
Using the Fuzzy Integral
HOSSEIN TAHANI, STUDFNT M E M B E R , i w r , ~ h n
JAMES M. KELLER, M E M R F R . r F r r
Abstract --Intelligent systems must be capable of integrating informa-
tion from a variety of sources. This information can be used to increase
object classification confidence, remove ambiguity inherent in a single
representation, and resolve conflict in separate decisions. Methods for
combining evidence produced by multiple information sources include
Bayesian reasoning, Dempster-Sharer belief theory, and heuristic mea-
sures of belief and disbelief. A method of evidence fusion, based on the
fuzzy integral, is developed. This technique nonlinearly combines objec-
tive evidence, in the form of a fuzzy membership function, with subjec-
tive evaluation of the worth of the sources with respect to the decision.
Various new theoretical properties of this technique are developed and
its applicability to information fusion in computer vision is demon-
strated through simulation and with object recognition data from for-
ward looking infrared imagery.
I. INTRODUCTION
Information fusion is an important aspect of any intelligent
system. The rationale behind bringing multiple input informa-
tion sources is that the information in any individual source is
either partial or contaminated, that is, it is uncertain and/or
imprecise. Pattern classifiers, scene analysis systems, image pro-
cessing systems, and computer vision systems all must be capa-
ble of integrating knowledge from various sources [ l]-[S].
The idea of representing a decision or a hypothesis in any of
the above paradigms is not new. A decision can be expressed as
an aggregation of a set of evidence that supports or rejects that
decision. Several methods for combining evidence produced by
multiple information sources have been studied by different
researchers; these include Bayesian reasoning, Mycin style mea-
sures of belief and disbelief, the Dempster-Shafer structure and
Manuscript received January 6, 1989; revised June I O , 1989 and October
12, 1Y89. This work was supported in part by Air Force Office of Scientific
Research grant AFOSR-X7-0226, and Emerson Electric research contract
AY8 188-25.
T h e authors are with the Electrical and Computer Engineering Depart-
ment, University of Missouri-Columbia, Columbia, M O 6521 1.
I E E E Log Number 8933534.
0018-9472/90/0S00-0733$01 .OO 01990 IEEE
MAY/JlINr I%)() 733
fuzzy logic [6]-[Y]. All these techniques have the ability to dcal
with conflicts in cvidcnce and to make predictions in the pres-
cncc of conflicting cvidence.
In this paper, a new evidence fusion technique, based on the
notion of the fuzzy integral is presented along with applications
to computer vision. The fuzzy integral differs from the above
paradigms in that both objective evidence supplied by various
sources and the expected worth of subsets of these sources are
considered in the fusion process. The fuzzy integral is a nonlin-
ear functional that is defined with respect to a fuzzy measure,
which in turn is either a belief or a plausibility measure in the
sense of Dempster-Shafer belief theory [lo]. The fuzzy integral
combines objective evidence for a hypothesis with the system’s
expectation of the importance of that evidence to the hypothe-
sis.
The systems that will be developed are represented as a
tree-like structure [11]-[ 131 where each knowledge source is
treated as a node. A node B is A’s child and A is B’s parent, if
B is directly below A in a tree. Each of the children in this tree
structure may also be a parent where its children are again
knowledge sources that support the belief or doubt in the parent
knowledge source, etc. In order to avoid triviality, we assume
that every node that is not a terminal node has more than two
children. Fig. 1 depicts a tree-like structure for a system that
uses such hierarchical evidence.
Section I1 develops fuzzy measures and their properties as
they relate to the information fusion problem. The fuzzy integral
and its place in decision making under uncertainty is also
presented there. Several new theoretical results are established.
In Section 111, the applications to feature level evidence combi-
nation and multisource fusion are described and the results of
applying this technique to automatic target recognition are
presented in Section IV. These results are compared to a
Bayesian decision structure and a Dempster-Shafer scheme,
highlighting the value of the fuzzy integral for this type of
evidence fusion.
11. TiiLcmt-i-icAL DEVELOPMENTS
Underlying the concept of the fuzzy integral is that of a fuzzy
measure. In this section we develop fuzzy measures and the
fuzzy integrals and their properties as they relate to the infor-
mation fusion problem. This development highlights the differ-
ences between probability measures, the key concept in Bayesian
reasoning, belief measures, the central idea in Dempster-Shafer
belief theory, and the fuzzy integral.
A. Ficzzy Measures and g,- Fiizzy Measure3
Fuzzy measures are the generalization of classical measures.
The notion of a measure in a Euclidean space is a natural
generalization of such elementary notions as the length of a line
segment, the area of a rectangle and the volume of a paral-
lelepiped. Still a more general concept of a measure in an
arbitrary abstract set can be defined [14], [lS].
Definition I : By a measurable space we mean a pair ( X , R )
consisting of a set X and a cT-algebra of subsets of X . A subset
A of X is called measurable (or measurable with respect to R )
if A E f l .
Definition 2: A measure p on a measurable space ( X , R ) is a
real nonnegative set function defined for all sets of R such that
~ (= 00,)and if {A,):-, is a disjoint family of sets with A , E R,
i > 1, then
 I

734 I E E E TRANSACTIONS ON SYSTEMS, MAN, A N I ) C.YHERNETJ('S,VOI.. 20, NO. 3, MAY/JUNE 1990

Fig. I . Tree structure for hierarchical evidence combination

It can be shown that a measure p has the following properties Then we can write [19],
[ 141: m m -I in

1 ) p ( A ) < p ( B ) if A c B .
g( A ) = c g', + A
I= I j = 1 k = j + l
g',g'L + . . . + A " l p ' g ' ~. . . gftn

2 ) If (Ai):=, is an increasing sequence of measurable sets,

then
=[ nA,€ A (l+Ag')-l]/A. A#O. (1)

Thus the value of A can be found from the equation

g(X)=l. (2)
An important example of such a measure is the probability This is equivalent to solving the equation
measure, P , where P ( X ) = 1. It

Within the framework of a human reasoning model, the A+l= n(l+Ag'). (3)
additivity hypothesis of this measure is too restrictive (see Sec- r=I
tion 111). In the seventies, alternative models were proposed by Hence if we know the fuzzy densities, g ' , i = 1; . ., n , we can
different researchers [7], [8], [161, [17] who all share the follow- construct the g,-fuzzy measure. For the information fusion
ing intuitively reasonable axioms. problem, we interpret the fuzzy density value g' as the degree of
Definition 3: Let g : R + [0,1]be a set function with importance of some source x , towards the final evaluation. The
measure of an arbitrary set A represents the importance degree
of the set of sources denoted by A towards a final decision. In
1) g ( 0 ) = 0, g ( X ) = 1 ,
2 ) g ( A ) d g ( B ) if A c B, [191, the following lemma characterizing the parameter A is
3) If ( AJ:=, is an increasing sequence of measurable sets, proved by induction. A simpler proof based on calculus is
then presented here.
Lemmu I: For a fixed set of [ g ' ) , 0 < g ' < 1 , there exist a
unique A E ( - 1, + =), and A # 0, which satisfies (3).

Proofi It suffices to show that the line A + 1 and polynomial

Such a function is called a fuzzy measure by Sugeno [16]. Note n
that g is not necessarily additive. Axiom 2 (monotonicity) is G(A)= n ( l + h g ' )
substituted for the additivity axiom of the measure. All belief i= 1
and plausibility functions (which includes probability measures) intersect only at one point for A > - 1 and A # 0. Now
are examples of fuzzy measures.
By the nature of the definition of a fuzzy measure g , the
measure of the union of two disjoint subsets cannot be directly
computed from the component measures. In light of this, Sugeno
[16] introduced the so-called g,-fuzzy measures satisfying the
following additional property: for all A , B c X and A n B = 0 , and I

g ( A U €3) =g( A) + g ( B ) + Ag( A ) g ( B ) , for some A > -1.

k=l r=l
r#k
A g , -fuzzy measure is indeed a fuzzy measure, and the g , -fuzzy
measure for A = 0 is a probability measure [18].
Thus G(A), G'(A) and G"(A) are all greater than zero for
B. Properties of the g,- F u u y Measures A > - 1 . That is, G is positive, increasing and concave up in the

-
open interval ( - 1 , m). Moreover,
Let X = ( X , ; . . , x r 1 )be a finite set and let g'
= g ( { x r ) ) .The n
mapping x , g' is called a fuzzy density function. Suppose G(O)=l and G'(O)= C g ' .
i= 1
A ={ x r y. ' ,XI,,,} G X . Note that the slope of the line A + 1 is equal to 1. In the case of

I I
 I

IEEE TKANSAC'TIONS ON SYSTEMS, M A N . A N D <'YHFKNFTIC'S, VOI . 20. NO. 3. MAY/JI'NT 1YYO 73s

+
C,g' < 1, G(A) crosses A 1 at zero with a slope of G'(0) < 1.
The fact that G(A) is a polynomial of at least sccond degree
increasing in the interval ( - l,m), it must cross A + 1 just oncc
again for some large enough A . A similar argument applies to
the case
If

Cg'>I.
r=l

This lemma makes the calculation of the parameter A much

easier. In fact, one needs only to solve an (n - I)st degree
polynomial and find the unique root greater than - 1.
Example 2: Consider the following simple case of three 0.0: . I . I . I . , . , i g5
0.0 0.2 0.4 0.6 0.8 1.0
knowledge sources, i.e., X = { x , , x , , x J together with density
values g ' =0.1, g 2 =0.3, and g"0.2. Using (3), the Sugeno Fig. 2. Behavior of g ( A ) when sum of densities is less than I .
measure g must have a parameter A satisfying 0.006A' 0.1 1 A -+
0.4 = 0. The unique root greater than - 1 for this equation is
A = 3.109, which produces the following fuzzy measure on the
power set of X (calculated from the definition of Sugeno
measures, or equivalently, (1)):
Subset A g3.,0v(A )

0
0.1
0.3
0.2
0.493
0.362
0.687
o.2 1
1.o
0.0 0.2 0.4 0.6 0.8 1.0
As expected, the subset of criteria { x z , x , ) is considerably more Fig. 3. Behavior of g ( A ) when sum of densities is greater than I .
important for confirming the hypothesis than either subsets
{ x , , x , } or { x , , ~ , ) .We will return to this example in Section
11-C. Several other more complicated examples are presented in For the case when A n B # 0 , experimental results suggest
Section IV. that g , ( B ) > g2(B). Fig. 2 shows the following situation. Let
We now establish a new result relating the parameters of two X = ( x , ; . . , x , ) be a set of sources. Assuming C:=,g'< I, g ' ,
g,-fuzzy measures. g', g', g4 and g h were fixed and g 5 was incremented. Then
g((x,,x,)) and g((x,,x,)) were plotted. As can be seen in Fig. 2,
Lemma 2: Let g, and g, be two g,-fuzzy measures with g({xl, x s } ) increased while g({x,, x , } ) decreased as g 5 increased.
parameters A , and A, respectively. Suppose g ; > gi for all i. This result is very intuitive and reasonable. As the the impor-
Then A , < A , . tance of source x , increases, one should expect that the mea-
sure (degree of importance) of any subset containing criteria x s
Proof Let G,(A)=FIY=,(l+Agi), j = 1 , 2 . Then the two should also increase, while those subsets not containing x s
functions G,(A) intersect the line A + 1 at 0 with slopes E:= ,g;, should decrease or certainly not increase in importance. Similar
j = 1,2 respectively. Now since G,(A) and G,(A) are increasing results are obtained when 1,s'> 1, as can be seen in Fig. 3.
and G,(A) > G,(A) for A > 0 and G,(A) < G,(A) for A < 0, it
follows that A , < A,, completing the proof. C. Belief and Plausibility Measures
Now, if The g,-fuzzy measures are a special subclass of belief and
plausibility measures defined by Shafer [7]. Let p ( X ) be the
(4) power set of X . Then a function bel: p ( X ) -+ [0,1] is a belief
function if and only if following hold [7]:
and gl is not greater than or equal to g i , for every i, the result I ) b e l ( 0 ) = O , b e l ( X ) = 1,
of the lemma in general, does not hold. Because of the inequal-
ity in the lemma, it is difficult to compare two g,-fuzzy mea-
2) bel( U A , ) >
I 0#/
6(1,..'.II)
( - 1)111+1bel( nI
A,).

sures, g , and g z on an arbitrary measurable space ( X , f l ) .

Hence, if it is only known that (4) holds, then g , ( A ) and g,(A) By duality, given a belief function bel, a plausibility measure,
cannot be compared. However, from the application point of PI, is defined for any subset A of X as follows [7]:
view, it is possible to categorize changes of a g,-fuzzy measure
when g({x,}) changes for some subset of the sources, as we do in PI ( A ) = 1 -bel ( x).
this proposition.
Some of the properties of bel and PI measures include:
Proposition I : Let g , and g, be two g,-fuzzy measures with
fuzzy densities {gi: i = 1,2;. . , n ) , k = 1,2. Suppose g j > g: for bel(A)+bel(x)<l, PI(A)+PI(x))l, and
x, E A X and gl = gl, x , E A . Then
PI ( A ) >/ bel ( A )
g,(B) < g,( B ) , if I3 A =O.
for all A E p ( X ) . Let g be a g,-fuzzy measure, then g is a
Proof: This follows from (1) and Lemma 2. belief (plausibility) measure if A > 0 ( A < 0 ) [18].
736 I E E E TRANSACTIONS ON SYSTEMS. M A N , A N I ) ('YHTKNKTI('S. VOI.. 20. NO. 3, M A Y / . I I J N K 1990

Example 2: The measure defined in Example 1 is a belief 3) If A c B , then

measure. For that same situation consider the set of density
values g ' = 0.313, g' = 0.638, g' = 0.507. Generating (3) and
solving for A gives A = -0.757, which generates the Sugeno
measure.
Subset A g-I, 757(A) In addition, we have established the following that is also casily
proved.
(XI) 0.3 13
(XJ 0.638 4) Let ( A , : i = l ; . . , n ) be a partition of the set X . Then
(XSl 0.507
(x,,x,} 0.8
Ix,,xs) 0.7
( x , , xs} 0.9
(x,,x,,xJ 1.0 where e , is the fuzzy integral of h with respect to g over A,.
The interpretation of all these properties related to the fuzzy
This measure is a plausibility measure, and, in fact is the dual integral as an information fusion technique should be obvious.
plausibility measure for the belief function generated in Exam- The calculation of the fuzzy integral when X is a finite set is
ple 1, that is, easily given [lo], [16]. Let X = { x , , x , ; . . , x , , } be a finite set and
g , 109(A)=1-g-1,757(~).
let h: X + [0,1] be a function. Suppose h ( x , ) > h ( x , ) 3 . . . >
h ( x , , ) ,(if not, X is rearranged so that this relation holds). Then
In this paper g,-fuzzy measures will be used extensively. a fuzzy integral, e, with respect to a fuzzy measure g over X can
These measures are easy to construct and they enjoy the proper- be computed by
ties of belief and plausibility measures.
D. Fuzzy Integrals e = max [ m i n ( h ( x , ) , g ( ~ , ) ) ] (5)
r=l
Using the notion of fuzzy measures, Sugeno [I61 defined the
where A , = ,,x!l.
concept of the fuzzy integral. Fuzzy are
functionals very similar to Lebesgue integrals, where the integral Note that when g 1s a g,-fuzzy measure, the values of g ( A , )
can be determined recursively as
is defined over measurable sets [14, 151.
Definition 4: Let ( X , C l ) be a measurable space and let
h : X + [ 0 , 1 ] be a Cl-measurable function. The fuzzy integral
over A c X of the function h with respect to a fuzzy measure g
is defined by
Thus the calculation of the fuzzy integral with respect to a
g,-fuzzy measure would only require the knowledge of the
density function, where ith density, g', is interpreted as the
degree of importance of the source x , for i = 1,2;..,n. Fur-
thermore, the degree of importance may be interpreted as a
belief if
where n
F, = (x : h(x ) 2 a } . Cg'<1,
1 x 1
The following is the interpretation of the fuzzy integral that
will be adopted in this paper. Suppose that an object is evalu- and a plausibility value if this sum is greater than 1.
ated from the point of view of a set of sources X. Let h ( x ) E [O, 11 Let h: X -+ [0,1] be a function and g a g,-fuzzy measure. Let
denote the decision for the object when source x E X is consid-
ered and let g ( ( x } ) denote the degree of importance of this
source. Now, suppose the object is evaluated using sources from
A c X. It is reasonable to consider a quantity
W ( A ) = min h( x ) = min ( h (x , ) , g ( A , ) ) for some 1 < I < n.
x € A
Define
as the best security decision that the object provides and g ( A )
expresses the grade of importance of this subset of sources. The if h( x ) > h( x , )
value obtained from comparing these two quantities in terms of
the min operator is interpreted as the grade of agreement f ( x ) = (Lix), otherwise.
between real possibilities, h( x ) , and the expectations, g. Hence
fuzzy integration is interpreted as searching for the maximal where e < c < 1. Then
grade of agreement between the objective evidence and the
expectation. jxf( x g( =e= jX h ( x 1 g ( .)
The following properties of fuzzy integral are easy to prove
[IO]. We now examine new results concerning the relationship
1) If h ( x ) = c , for all x E X , 0 < c < 1, then between the fuzzy integral of a function h when the integral is
taken with respect to two g,-fuzzy measures, g,,g,. First, note
that if g , and g, are related by g , ( A ) + g2(&= 1, that is, g ,
and g , are the corresponding belief and plausibility measures,
then the fuzzy integral of a function h with respect to a belief
2) If h l ( x ) < h , ( x ) for all x E X then measure is less than or equal to the fuzzy integral of h with
respect to the plausibility measure. This argument establishes
the following proposition.

7
 I

IEEC TRANSACTIONS O N SYSTFMS, M A N , ANI> ('YHFKNCTIC'S, VOI.. 20. NO. 3, MAY/JIJNF 1990 737

Y Function h
Initial Measure
1.0-
Final Measure

0.8 -
0.6 -

0.4 -
0.2 -
0.04 . , . , . , . , . , . , . , . , ,
0 1 2 3 4 5 6 7 8
Fig. 4. Behavior of fuzzy integral when g 7 i s increased from 0.1 to 0.4.

1.0-
- I Function h
Initial Measure
Final Measure

0.8 -
0.6 -
0.4 -
0.2 -
o.o!
0
. ,
1
. I
2
.
3
, . , .
4
1

5
.
6
, .
7
. ,
8
.
Fig. 5. Behavior of fuzzy integral when g' is increased from 0.05 to 0.4.

Proposition 2: If g , and g, are two g,-fuzzy measures related
by g , ( A ) + g,(x)=1, A L X,with A , > 0, then
Using Proposition 1, the behavior of the fuzzy integral when
g' changes for some 1 < i d n can now be described. First, note
that if g' increases then g ( A , ) , where A , ={x,;..,x,}, i =
2; . .,n , also increases and hence the new fuzzy integral must be
greater than or equal than the previous fuzzy integral. In gen-
eral, if g k increases for some I < k < n , then g ( A , ) , i = 1; . .,
k - 1, decreases and g ( A , ) , i > k increases (see Proposition 1).
Now, suppose
suppose that h ( x , < h ( x , ) . That is, x A has lesser objective
evidence in supporting the hypothesis or decision than x I . Now,
if g h increases, then by the previous argument, g ( A , ) , i =
1 , . . ., I , . . . ,k - 1, will decrease and hence the new fuzzy integral
will be smaller than or equal to the previous fuzzy integral.
Similarly, if h ( x , ) > h ( x , ) and g k increases, the new fuzzy
integral becomes greater than or equal to the previous fuzzy
integral.
Now, if two densities g h and g"' for 1 < k < rn B n increase,
then g ( A , ) , i = l ; . . , k -1, will decrease and g ( A , ) , i =
k , . . .,in,. . . ,n will increase. Thus, the fuzzy measure would
change corresponding to the cases considered previously and
hence the fuzzy integral would also change accordingly. These
results are intuitively very satisfying. That is, if the function h or
measure g increases then the integral increases. Also, if some
source becomes more important (more believable or plausible)
in later stages of processing, and if the objective evidence
regarding the source has a lower value than the source which
generated the integral, then the integral (overall evaluation)
would decrease. Fig. 4 shows the behavior of the integral when
the degree of importance of some x k (here x,) increases for
h ( x , ) < h ( x , ) . Fig. 5 shows the same result when h ( x k )> h ( x , ) ,
for k = 3. In both figures, the labeling on the horizontal axis has
two interpretations. In plotting the function h , each index i
corresponds to the appropriate source x , . With respect to the
fuzzy measures, the index i corresponds to the set A , =
{I,; . ., x,). In this way, the fuzzy integral can be read from the
plot as the intersection of the graph of h with that of g .
111. APPLICATIONS
The fuzzy integral was used as a segmentation tool in [20],
[21]. Here, the design and implementation of an object recogni-
tion system using the fuzzy integral is explained. The output of
this system can be considered as a decision, or a hypothesis for a
higher level of recognition.
A. Classification Integration
In many cases an object can be represented as a vector in an
n-dimensional Euclidean space, where each component of this
vector is a feature measured from that object. There are many
different types of features that can be calculated from objects,
e.g., shape measures, texture measures, and statistical measures,
to name a few [ I , 21.
The reason for measuring different features is that there is
usually no single feature that can identify the objects of interest.
In fact, there is normally no set of features that always distin-
guishes one object from others precisely. There is always an
uncertainty inherent in the recognition problem. Instead, each
feature or group of features can be considered as evidence in
the identification of an object. Obviously, each of these features
or group of features would have a degree of importance in the
identification of an object. For example, consider the identifica-
tion of a person by using his eyes or his forehead as a mean of

------ 7 1
738 IEFF. TRANSACTIONS O N SYSTEMS, M A N , A N D ('YHF.RNFTI<'S, V o l . ,
recognition. Certainly, the feature sets x I = (eyes), x 2 =
(forehead) and x 3 = (eyes, forehead) have different degree of
importance in the recognition of the person even though each
have different partial evaluation for the identification. More-
over, the importance of xi is not necessarily the sum of the
degree of importances of x I and x 2 .
There are many algorithms that can be used for the partial
identification purposes using the features calculated from a
region. These include algorithms based on Bayes classifiers,
nearest prototypes, perceptrons, K-means, fuzzy K-nearest
neighbors, fuzzy perceptrons, etc. [6], [22]-[24].
In the case of fuzzy algorithms, the result of the identification
is a number in the closed interval [O, 11. For nonfuzzy algorithms,
however, the result of the identification is either 0 or 1. In the
case of the Bayes classifier, the U posteriori probability gener-
ated by this algorithm can be taken as a degree of identification.
Then these numbers are considered as partial evaluations of the
objects. Each of these evaluations may support or contradict one
another.
In any case, the identification problem is as follows: Let
T =(tI; ..,t,,,) be a set of classes of interest. Let A be an object
in the scene. Is object A in class t, for 1 < i < n? That is, with
the given evidence, what is our belief that object A is in class t,?
Formally, let T , as defined before, be the set of classes of
interest. Note that each t , may, in fact, be a set of classes by
itself. Let X = (x,, xZ; . ., x,,) be a set of groups of features or
algorithms for the recognition of a particular class, t , , 1 < i < n .
Here, a group of features or algorithms also includes a single-
ton. Let A be the object under consideration for recognition.
Let h,: X +[0,1] be the partial evaluation of the object A for
class t,, that is, h , ( x , ) is an indication of how certain we are in
the classification of object A to be in class t k using the group x,
where a 1 indicates absolute certainty that the object A is really
in class tk and 0 implies absolute certainty that the object A is
not in t k . Now corresponding to each x, the degree of impor-
tance, g', of how important x i is in the recognition of the class
tk must be given. These densities can be subjectively assigned by
an expert, or can be generated from training data [21]. The g"s
define the fuzzy density mapping. Hence A can be calculated
using (3) and the g,-fuzzy measure, g can be constructed. Now,
using (5) and (6), the fuzzy integral can be calculated. Thus the
following algorithm for object recognition at this level, referred
to as classification level integration or fusion is given.
BEGIN class-finder: classification level
DO for each class of interest t ,
DO for each group of features or algorithm x
calculate h , ( x )
determine g k ( ( x ) )
E N D DO
calculate A, from (3)
compute fuzzy integral from ( 5 ) and (6)
E N D DO
END class-finder classification level
If the goal is to make a decision at this level, the class t , with
the largest integral value can be chosen. However, fuzzy inte-
grals as calculated previously would normally be considered as
the partial evaluations for the next higher level of recognition.
B. Multisource Integration
Now, assume that the scene under consideration is analyzed
by different methods. This could include analysis over time, or
different sensor types, e.g., radar, forward looking infrared
(FLIR), day television (TV), laser range finder, etc. The systems
can automatically detect objects of interest within their search
area. They can also measure features associated with the ob-
jects.
1
20, NO. 3. M A Y / J U N F 19%)
Using the aforementioned method, the evidence at the classi-
fication level can be combined to obtain a partial evaluation for
the object at the source level. Now each of these sources has a
different degree of importance in the recognition of the classes.
That is, each subsystem, as a source of information, gives
evidence supporting or rejecting the existence of the object in
the scene constrained by the fact that, still at this level, identifi-
cation of thc object is uncertain. For cxample, a TV sensor,
under some cnviromcntal conditions (like at twilight), would
give more information than a FLIR sensor. These bodies of
evidence can be integrated at the sensor types for the whole
system.
The output of the system would again be a number in the
closed interval [0,1] for each class of interest. These numbers
can be used to derive a decision or validate a hypothesis for the
existence of class t, for 1 < i < n , depending upon the configura-
tion of the system.
Finally, at each instance of time, different sensors could
observe the scene under consideration. This gives rise to a
hicrarical decision structure where the evidence at the sensor
level can be combined to get a degree of existancc for a
particular class for each frame and then these bodies of evi-
dence can be integrated over time to arrive at a hypothesis or a
decision for the object under consideration. Thus, the following
algorithm will accommodate any case discussed above. The
difference between this algorithm and the one for classification
level fusion is the calculation of h , in the sense that the values
of h can be the fuzzy integral values calculated in a lower level
of recognition.
BEGIN class-finder: multisource fusion
DO for each class of interest t ,
DO for each information source
Invoke class-finder classification level to
determine g,((x))
E N D DO
calculate A, from (3)
compute fuzzy integral from (5) and (6)
t N D DO
LND class-finder
Here again, the fuzzy integral gcneratcs a mapping, for each
object of interest, from the set of classes to the closed interval
[U, 11.
IV. R t s u L n
The results presented herc were produced by software implc-
mentation of the fuzzy integral algorithm developed in Section
111. The software was developed using Fortran77 in the
Computer Vision Laboratory at the University of Missouri-
Columbia.
The test data consisted of simulation data and real imagery
from an automatic target recognition problem. The simulation
data was used to highlight the behavior of the fuzzy integral
under controlled conditions. This data could simulate any of the
situations described above. The real data consisted of several
sequences of FLIR images containing an armored personnel
carrier (APC) and two different types of tanks. A preprocessing
step was run on each image to detect objects of interest, and
features were calculated for each of these objects.
A. Results -Sirnulution Duta
The behavior of the fuzzy integral of a function h with respect
to a RA-fuzzymeasure, g is cxamined. First, assume that h is a
constant function, that is, h = c , 0 < c < I . In this case, the fuzzy
integral is also c no matter what fuzzy measure is used (see
Section IV). This is because g ( X ) = 1. Suppose now that g is
fixed and h varies. Since the values of h arc rearranged to
calculate the fuzzy integral, the behavior of the fuzzy integral
would be similar to the case of a fixed h and modified g. Table I
I E E E TRANSACTIONS ON SYSTEMS. M A NA. N I ) C . Y B E R N E T I ( ' S ,V O I . 20, NO. 3, M A Y / . I I J N F 1990 739

TABLE I TABLE 11
VALUES
OF THEFuzzy INTEGRAL FOR D I F F E R E N T OF FUZZYINTFGRAI., BAYES
COMPARISON A N D DEMPSTER-SHAFER
DENSITIESFUNCTIONS" FOR Two CLASS
CI.ASSIFIERS PROBLEM
A
A
C a s e i 1 2 3 4 5 6 ~
h THEREsu1.T OF T I I E FEATURE FUZZYINTEGRAL
LEVEL.
Cornpitted densities and A ralires
1 g' 0.2 0.4 0.6' 0.4 0.3 0.15 0.60 -0.904 g1 6' g3 g4 h
2 g' 0.6 0.6" 0.7 0.5 0.8 0.4 0.80 -0.997
3 g' 0.1 0.3 0.2'0.35 0.2 0.1 0.55 -0.448 Tank 0.16 0.23 0.19 0.22 0.760
4 g' 0.4 0.4' 0.8 0.4 0.3 0.15 0.65 -0.969 APC 0.15 0.24 0.18 0.23 0.764
5 g' 0.6 0.4' 0.8 0.4 0.3 0.15 0.76 -0.980
6 g' 0.2 0.4 0.6' 0.4 0.8 0.9 0.60 -0.997 Confusion Matrix
7 g' 0.05 0.1 0.4' 0.6 0.8 0.9 0.48 -0.996 Tank APC
h ( x , ) 1.0 0.8 0.6 0.45 0.3 0.1
Tank 175 1
"When sum of densities is greater than 1. APC 17 49
'Indicates the source which defines the fuzzy integral value. Total correct 92.6%.
B
B
C a s e i 1 2 3 4 5 6 e h T H ERESULT
OF THE BAYES
CLASSIFIER
Confirsion Matrir
1 g' 0.01 0.04 0.02 0.01 0.06" 0.05 0.30 24.755 Tank APC
2 g' 0.05 0.05 0.08 0.1' 0.1 0.8 0.42 4.111
3 g' 0.005 0.01 0.01 0.005 0.04" 0.03 0.29 75.389 Tank 176 0
4 g' 0.1 0.04 0.08 0.02' 0.01 0.06 0.45 10.344 APC 22 44
5 g' 0.2 0.04 0.08' 0.02 0.01 0.06 0.53 6.663 Total correct: 90.91%.
6 g' 0.01 0.04 0.02 0.01 0.06' 0.2 0.24 11.550
7 g' 0.005 0.02 0.01 0.01 0.05' 0.3 0.14 13.180 C
h ( x , ) 1.0 0.8 0.6 0.45 0.3 0.1 T H ERESULT
OF THE DEMPSTER-SIIAFER
CLASSIFIER
"When sum of densities is less than 1. Confusion Matrir
'Indicate the source that defines the fuzzy integral value. Tank APC
Tank 160 16
APC 17 49
shows these results. Here, a partial evaluation function h is Total correct: 86.36%.
fixed and the g,-fuzzy measure g is changed. Observe that, in
general, the fuzzy integral with respect to a plausibility measure
(Table I-A) is a larger number than the fuzzy integral with sponding to the fuzzy integral value. This is, in fact, the way that
respect to a belief measure (Table I-B). This is due to the an information fusion technique should perform.
restriction that these measures impose on the fuzzy densities. B. Results -Real Data
Each case shows a set of fuzzy densities corresponding to six
sources and the fuzzy integral value of h with respect to the The algorithms stated in Section I11 were tested using the
measure obtained by these fuzzy densities. Moreover, the source FLIR data. There were three sequences of 100 frames each
that defines the fuzzy integral value and the value of A are used for training purposes. In each sequence the vehicles ap-
given. In case 1 an arbitrary set of fuzzy densities is given. Cases peared at a different aspect angle to the sensor (0",45",90"). In
2 and 3 represent the cases when all densities are increased and the fourth sequence the APC "circled" one of the tanks, moving
decreased respectively. In both cases the fuzzy integral values in and out of a ravine, and finally coming toward the sensor.
were increased or decreased correspondingly. In cases 4 and 5 , This sequence was used to perform the comparison tests. The
the densities corresponding to sources 1 and 3 were increased images were preprocessed to extract object of interest windows.
with the density corresponding to source 1 increased more in The classification level integration was performed using four
situation 5 than it was increased in case 4. As expected, the statistical features calculated from the windows. To get the
fuzzy integral followed this rise in expectation for both the belief partial evaluation, h ( x ) , for each feature, the fuzzy two-mean
and plausibility measures. In case 6, the fuzzy densities corre- algorithm [22] was used. The fuzzy densities, the degree of
sponding to the source with smaller h values (objective evi- importance of each feature, were assigned based on how well
dence) were increased. In the case of the belief measure the these features separated the two classes Tank and APC on
integral value decreased and in the case of the plausibility training data. The algorithm for doing this follows:
measure the integral value did not change. Hence, with in- Let (ftr1,ftr2; . .,fir,,} be a set of features and (C,; . .,C,,,} be
creased importance on those information sources for which a set of classifiers. Consider the n x rn matrix P with p , , to be
there is little objective evidence, the fuzzy integral does not the performance of classifier C, using feature ftr, for the partic-
increase, and in general, will decrease. Situation 7 shows that ular class or hypothesis under consideration. Let
when the densities corresponding to the sources with greater sum = cp,,.
objective evidence than the one corresponding to the integral r.j
value decrease and the densities corresponding to the sources
with smaller objective evidence than the one corresponding to Define
the integral value increase, the integral value will decrease. p , , .d sum
However, this is not always the case. This really depends on the b,, = ~,
sum
function h and the amount of change in the density function.
Once the source corresponding to the fuzzy integral value is where d s u m is the desired sum of fuzzy densities. Then the
known, we may distinguish each source as one with greater or computed fuzzy density for the ith feature is given by
smaller objective evidence. When the density function changes, !n
the new fuzzy integral value will change depending upon how
these changes are balanced with respect to the source corre-
g'= c 4,.
,= I

7
 I

740 l r r K T K A N S A C T I O N S O N XYSTTMS. M A N , A N I ) ('YI$ERNrTI('S,V O L . 20. N O . 3, MAY/JI!NF 1990

TABLE 111
~~~~~~ ~~ ~ ~ ~

A
THERESULTS
OF I N F O R M A T I O N FUSION
USING T I I E Fl!ZZY INTECIKAI O N TIIREF CI.ASSIFIEK1, FOR TANK
DIFFFKENT

Actual Partial evaluation for Tank Fuzzy Integral

Object Fuzzy Feature level Evaluation for
Class Bayes K-Mea:: Fuzzy integral Tank Hypothesis
~ ~

1) Tank 1.oo 0.77 0.68 0.68

2) Tank 1.oo 0.85 0.71 0.71
3) Tank 1.oo 0.81 0.71 0.71
4) Tank 1.oo 0.83 0.7 I 0.7 1
5 ) Tank 1.oo 0.76 0.71 0.71
6) APC 0.44 0.44 0.40 0.40
7) APC 0.00 0.27 0.27 0.27
8) APC 0.00 0.53 0.49 0.43
9) APC 0.99 0.26 0.25 0.26
IO) APC 0.97 0.18 0.21 0.21
B
THERESUI.TOF INFORMATION
FUSION
USINGTHE FUZZYINTEGRAL.O N
THREE DIFFERENT FOR APC
CLASSIFIERS

Actual Partial Evaluation for APC Fuzzy Integral

Obiect Fuzzy Feature Level Evaluation for
class Bayes K-Mein Fuzzy Integral APC Hypothesis
1) Tank 0.00 0.23 0.32 0.23
2) Tank 0.00 0.15 0.28 0.20
3) Tank 0.00 0.19 0.29 0.20
4) Tank 0.00 0.17 0.28 0.20
5 ) Tank 0.00 0.24 0.27 0.24
6) APC 0.56 0.56 0.55 0.55
7) APC 1.00 0.72 0.73 0.72
8) APC 1.oo 0.49 0.47 0.47
9) APC 0.01 0.75 0.65 0.33
10) APC 0.03 0.82 0.65 0.33

The result of the fuzzy integral classifier is presented in the
form of confusion matrix, in Table 11-A, where the count of
samples listed in each row are those which belong to the
corresponding class and the count of samples listed in each
column are those after classification, which was made by choos-
ing the class with largest integral value.
A Bayes classifier was trained on the same four features from
the three training sequences and tested on the fourth. This
classification, based on largest posterior probability is given in
Table 11-B. As can be seen, the overall classification rate for the
fuzzy integral was higher than that for Bayes. In addition, the
final integral values provide a different measure of certainty in
the classification than posterior probabilities. The integral eval-
uation need not sum to one, so that lack of evidence and
negative evidence can be distinguished.
These algorithms were also compared to a Dempster-Shafer
classification scheme. In this experiment, the fuzzy two-means
was used on each of the same four features to generate two
simple support functions for the two classes. These belief func-
tions were then combined using Dempster's Rule [7]. The ob-
jects (fourth sequence) were classified into the class that had the
largest belief. Table 11-C contains the confusion matrix for this
classification scheme. Again, the fuzzy integral outperformed
the Dempster-Shafer algorithm in this experimental setup.
A conceptual difference between the fuzzy intcgral and a
Dempster-Shafer classifier is in the frame of discernment. For
the fuzzy integral, the frame of discernment contains the knowl-
edge sources related to the hypothesis under consideration,
whereas with belief theory, the frame of discernment contains
all of the possible hypotheses. Thus the fuzzy integral algorithm
has a means to assess the importance of all groups of knowledge
sources towards answering the question as well as the degree to
which each knowledge source supports the hypothesis. With
belief theory, each knowledge source would have to generate a
belief function over the power set of the set of hypotheses, with
these combined using Dempstcr's rule. This calculation has
exponential complexity with the number of hypotheses. With
the fuzzy integral, the Sugeno measure need only be calculated
for n subscts (where n is the number of knowledge sources) for
each hypothesis. These measures are then combined with the
objective evidence to produce the integral values.
Finally, three classifiers, the fuzzy integral, the Bayes classi-
fier and the fuzzy K-Mean were used to represent different
sources. The posterior probabilities obtained from the Bayes
classifier together with classification results from fuzzy integral
and the fuzzy K-Mean were taken as partial evaluations for the
objects of interest in a multisource integration. The degree of
importance of each source again was based on how good these
classifiers performed on a training data set. Then these bodies
of evidence were combined using the fuzzy integral. The results
are in Table 111. The results show that this methodology pro-
duced good estimates of class confidence based on the objective
information and the subjective expectation of the importance of
the information. As can be seen, objects 9 and 10 were misclassi-
fied by the Bayes algorithm. However, in the final evaluations
they were correctly classified. The effect of misclassification by
Bayes has given rise to small fuzzy integral values for the APC
hypothesis in both cases. This information can be used by an
intelligent monitor to initiate more sophisticated procedures to
measure confidence in class membership.
V. CONCLUSION
In this paper, a methodology for information fusion using the
fuzzy integral was introduced. This method is based on tree-like
hierarchy of evidence, where each child node is a knowledge
source that provides a body of evidence to its higher-level parent
node. The parent node combines thc evidence provided based

-T 7

IFF-[ T K A N S A C ‘ T I O N S O N S Y S T E M S . M A N . A N I ) < . Y l i r K N F T I < ’ S , VOI
o n the degree of support and the relative importance of each
child. The most important advantage of this system is that not
only is the evidence combined but that the relative importance
of the different sources is also considered.
T o use the fuzzy integral as an information fusion technique
in a system, it is necessary to understand the behavior of the
fuzzy integral when the g,-fuzzy measure changes. In this paper,
theoretical and experimental results have been derived that
allows the prediction of the effects of changes in importance of
nodes to the overall evaluation. The results obtained are intu-
itively reasonable for information fusion purposes.
The fuzzy integral algorithm was applied to image data in an
automatic target recognition situation. The algorithms per-
formed well at both the feature level and source level integra-
tion. The fuzzy integral provides a natural coupling of objective
evidence and expectation.
REFERENCES
A. Rosenfeld and A. C. Kak, Digirul Picrrirr I’rocrxsmg. New York:
Academic Press, 1976.
D. H . Ballard and C. M. Brown, Coinpiirer Vision. Englewood. NJ:
Prentice-Hall, IYX2.
E. M. Riseman and A. R. Hanson, “ A Methodology for the develop-
ment of general knowledge-based vision systems.” Proc. I E I X Work-
shop on Principle of Knowledge-bused Sysriwis, Dec. IYX4, pp. I S Y - 170.
A. R. Hanson and E. M. Riseman. “VISION: A Computer System for
Interpreting Scenes,” in Coiprirer Vision Sysrmis, A. R. Hanson and
E. M. Riseman, Eds. New York: Academic Press, 197%
B. E. Flinchbaugh and B. Chandrasckaran, ”A theory of spatio-tem-
poral aggregation for vision.” Arrificiul Inre//., vol. 17. pp. 3x7-407.
T. J. Tou and R. C. Gonzalez, Purrern Recognition Principles. Reading,
MA: Addison Wesley, I Y X I .
G. A. Shafer, A Mu/hetnuricul Theor) of Ei.iderrce. Princeton. NJ:
Princeton Univ. Press, lY76.
L. A. Zadeh, “Fuzzy sets as a basis for a theory of possibility,” / n r . J .
Fiizz)’ Sers Sysr., vol. 1, no. I , pp. 3-2X, lY7X.
E. H. Shortliffc, Cornpurer-Bused Mcdicul Consriltorions: My<.in. New
York; Elsevier, IY76.
S . T. Wierrchon, “On fuzzy measure and fuzzy integral,” Fuzzy Infor-
inuriun und Deckion Processes, M. Gupta and E. Sanchcs, Eds. New
York: North-Holland, 19x2, pp. 71)-X6.
E. Charnik and D. McDermott, Inrrodricrion to Ai-rificiul Inrc4igmce.
Reading, MA: Addison Wesley, IYXh.
P. Winston, Arrifiriul Inrelligence. Reading, MA: Addison-Wesley.
lY7Y.
A. Barr and E. Feigenbaum, The Hundhook of Arrificiul Inrel/i!w7ce.
vol. 1. Los Altos, CA: Kaufman, 1 Y X I .
P. R. Halrnos Measure Theoty. New York: Van Nostrand, lY50.
W. F. Pfeffer, Inrequls und Meusures. New York: Marcel Dekker.
1977.
M . Sugeno, “Fuzzy measures and furry integrals: A survey,” Frizzy
Aiirornuru und Decision Processc~s. Amsterdam: North Holland. 1Y77.
pp. XY- 102.
T. Terano and M. Sugeno, “Conditional fuzzy meawres and their
application,” In Fuzz)‘ Aiirornuru u r d Tlzrir Applicurions IO Cognirirv und
Decision Processes. New York: Academic Press, IY75, pp. 151-170.
G . Banon, “Distinction between several subsets of fuzzy measures,”
Fr12q Sers S W . , vol. 5, I Y X I , pp. 291-305.
K. Leszczynski, P. Penczek, and W. Grochulskki. “Sugeno’s furry mea-
sures and fuzzy clustering,” Frizzy Sets Sysr., vol. 15. pp. 147-1%. IYXS.
J. Keller, H. Qiu, H . Tahani, “Fuzzy integral and image segmcntation.”
in Proc. Norrli Americun Ftizz)‘ Inforinurion Proctwing Soc... New Or-
leans, June IYXh, pp. 334-338.
H . Qiu and J. Keller, “Multispectral image segmentallon using furry
techniques,” in Proc. Norrh Arnericun Frizzy Infor7niiriorr h-occ~s.sirrgSoc..
Purdue University, May l Y X 7 , pp. 374-3x7.
J , C. Bezdek, Purrern Recogninon wirh Fuzzy Objcc/ii,e Frinctrorr A l w -
rirhms. New York: Plenum, I Y X I .
J. Keller. M. Gray, and J. Givens, “A fuzry K-nearest neighbor algo-
rithm,” IEEE Trurts. Sysr. Mun Cyhern., vol. 15. no. 4, pp. SXIl-5XS.
IYXS.
J . Keller and D. Hunt, “Incorporating furry membership functions into
the perceptron algorithm,” lEEE 7i.uns. Piirrrm Anul. Miicliitw InWIl..
vol. 7, no. 0,pp. 693-6YY, 19x5.
0018-9472/90/0500-0741$01.00 01990 IEEE
I
. 20. N O . 3 . MAY/.IIJNI 1990 74 1
Comments on “An Optimal Multiple Threshold
Scheme for Image Segmentation”
HAN LEE A N I ) RAE-HONG PARK, MFMHFR. irrr
Abstract-The paper hy Reddi etal. presented a fast search scheme
for finding single and multiple thresholds as a speed improvement to
Otsu’s scheme. The correspondence points out explicitly that their fast
search procedure doesn’t converge to the optimal threshold if the
interclass variance has not the unique maximum. The limitations of
Otsu’s technique are also discussed briefly.
Otsu [ 11 proposed a threshold selection method from his-
tograms based o n maximizing the scparability of the resultant
classes in gray levels. For a bimodal distribution case, the
interclass variance a ’ ( k ) is defined as
= p,,rnZ + p,mi - rn; (11
where p n is the probability of dark pixel whose gray level is less
than k , p B is the probability of bright pixel whose gray level is
greater than k , m,, is the mean of dark pixels, m, is the mean
of bright pixels, and m , , is the total mean. We can determine
the optimal threshold k * that maximizes 0 2 ( k ) . It is also
straightforwardly extended to multiple thresholding problems.
However, the maximization procedure becomes more and more
complicated computationally and even impossible to implement
practically as the number of thresholds to be selected increases.
Thus, Reddi et al. [ 4 ] proposed a fast search scheme to
overcome this limitation of Otsu’s method with an assumption
that a ’ ( k ) has the unique maximum. They attempted to reduce
the computation time equivalently by finding k such that m, +
m, = 2 k . However, the rigorous proof of the unimodality has
not yet been obtained [ l ] . Also Kittler and Illingworth demon-
strate that this conjecture of unimodality does not hold in
general [ 2 ] . Kittler and Illingworth show that the criterion fune-
tion may not only be multimodal, but more importantly, if it is
multimodal, its global maximum is not guaranteed to give eor-
reet segmentation results. The purpose of this correspondence is
to point out explicitly with experimental evidence that the speed
improvement by Reddi et al. fails to converge to the global
maximum, since Otsu’s method and the method by Reddi et al.
arc not equivalent mathematically, especially for cases with
a ’ ( k ) nonunimodal.
According to several experimental results, the proposed
method by Reddi et al. is often influenced by the local maxima
or minima of the interclass variance. In that case, the selected
threshold k is varied by changing the initial starting point. It is,
therefore, obvious that the fast search procedure by Reddi et al.
doesn’t converge to the same optimal threshold k * which maxi-
mizes a ? ( k ) . Mathematically, the optimal threshold k * for
which o ’ ( k ) is maximum is not equal to the gray level k such
that 6 a 2 ( k ) / 6 k = 0 which gives local maximum, local minimum,
or inflection points also. Also this is true for the multiple
t hresholds.
Manuwript received September 1 I, IYX7; revised March IX, IYXY. and July
X, IYXY. This wwrk was supported in part by the Korea Science & Engineer-
ing Foundation.
H . Lec is with thc C&C Lab., Telecommunication RXrD Div., Lucky-
GoldStar R & D Complex. Anyang 553, Hogae-Dong, Anyang-Shi, Kyongi-Do.
430-OXO, Korea.
R. H . Park is with the Department of Electronic Engineering, Sogang
University, C.P.O. Box 1142. Seoul I(H)-hI I , Korea.
I E E E Log Number XY34162.