Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp.

1837-184

STATISTICAL PRINCIPLES OF SOURCE SEPARATION

Jean-Francois Cardoso 
 ENST/CNRS, 46 rue Barrault, 75634 Paris. France.

http://sig.enst.fr/ cardoso/stuff.html

Abstract: Blind signal separation (BSS) is an emerging signal processing technique, aiming
at recovering unobserved signals or `sources' from observed mixtures (typically, the output
of an array of sensors), exploiting only the assumption of mutual independence between
the signals. The weakness of the assumptions makes it a powerful approach but requires
to venture beyond familiar second order statistics. The objective of this paper is to review
some of the approaches that have been recently developed to address this exciting problem,
to show how they stem from basic principles and how they relate together.

Keywords: Blind source separation, independent component analysis, contrast functions.

1. INTRODUCTION This is compactly represented by the mixing equa-

tion

Blind signal separation (BSS) consists in recov-

ering unobserved signals or `sources' from several
observed mixtures. Typically, the observations are
obtained at the output of a set of sensors, each
sensor receiving a dierent combination of the
`source signals'. The adjective `blind' stresses the
fact that i) the source signals are not observed
and ii) no information is available about the mix-
ture. This is a sound approach when modeling
the transfer from the sources to the sensors is
too dicult; it is unavoidable when no a priori
information is available about the transfer. The
lack of a priori knowledge about the mixture is
compensated by a statistically strong but often
physically plausible assumption of independence
between the source signals. The so-called `blind-
ness' should not be understood negatively: the
weakness of the prior information precisely is the
strength of the BSS model, making it a versatile
tool for exploiting the `spatial diversity' provided
by an array of sensors.
The simplest BSS model assumes the existence
of n independent signals s1 (t); : : : ; sn (t) and the
observation of as many mixtures x1 (t); : : : ; xn (t),
these mixtures being linear and instantaneous.
x(t) = As(t) (1)
where s(t) = [s1 (t); : : : ; sn (t)]y is an n  1 column
vector collecting the source signals, vector x(t)
similarly collects the n observed signals and the
square n  n `mixing matrix' A contains the
mixture coecients.
The BSS problem consists in recovering the source
vector s(t) using only the observed data x(t), the
assumption of independence between the entries
of the input vector s(t) and possibly some a priori
information about the probability distribution of
the inputs. It can be formulated as the compu-
tation of an n  n `separating matrix' B whose
output y(t)
y(t) = Bx(t) (2)
is an estimate of the vector s(t) of the source
signals.
The basic BSS model can be extended in several
directions. Assuming for instance more sensors
than sources, complex signals and mixtures and
noisy observations, one obtains the standard nar-
row band array processing/beamforming model.
Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp. 1837-184

Another extension is to consider convolutive mix-
tures: this results in a multichannel blind deconvo-
lution problem. All these extensions are of prac-
tical importance, but this paper is restricted to
the simplest model: real signals, as many sensors
as sources, non convolutive mixtures, noise free
observations because it captures the essence of the
BSS problem.
the strongest assumptions corresponds a priori
to the narrowest applicability. However, well de-
signed approaches are in fact surprisingly robust
even to gross errors in modeling the source distri-
butions as shown below. For simplicity, zero-mean
sources are assumed throughout: Es = 0.
Possible approaches to source separation are to
adjust a separating matrix in order to restore a
`distributional property' of the output y: maximal
peakedness or minimum entropy; to restore the
mutual independence of the outputs; or possibly
the joint distribution of all the sources. It is an
objective of this paper to show how these heuristic
2.2 Blind identiability
There are three levels of indetermination in the
BSS problem. One level is due to the data them-
selves: if, for instance, two sources have the same
distribution, they can only be recovered only up a
permutation. The second level may be due to lack
ideas dene `contrast functions', how they can be of prior information. If, for instance, the distri-
derived from rst principles and how they relate bution is known up to scale, this scale cannot be
to each other.
2. MODELING AND IDENTIFIABILITY.
2.1 Statistical modeling
Source separation exploits primarily the `spatial
identied because it could be represented as well
rescaling the corresponding column of A. More
generally, if the source distributions are unknown,
the best that can be done is to recover a copy
of the signals, that is the source signals up to
scales, signs and permutations; see e.g. Tong et
al. (1991a).
Achieving signal copy is sucient as far signal
diversity', that is the fact that several sensors separation is concerned: the rst two levels of
carry dierent mixtures of the sources. It often indeterminations can be considered harmless. The
ignores any time structure. In this perspective, the real issue is to avoid any wider indetermination:
BSS problem reduces to the identication of the
probability distribution of a vector x = As given
a sample distribution. Thus, the statistical model
has two components: the mixing matrix A and the
probability distribution of the source vector s.
Mixture. The mixing matrix A is the parameter
of interest. Its columns are assumed to be linearly
Gaussian signals can be recovered only up to a
rotation, which is unacceptable, of course. A key
result is given (for non-deterministic signals) by
Comon (1994) after a theorem of Darmois. In
essence: if there is at most one Gaussian source,
signal copy is equivalent to restoring statistical
independence between the signals. Threfore BSS
independent. There is something special about
having an invertible matrix as the unknown pa-
rameter, because the set of all n  n invertible
matrices forms a multiplicative group. This sim-
ple fact has a profound impact on source sep-
may be implementing by nding a separating
matrix such that y = B x has independent entries.
Using second order information. Even though sec-
ond order information (decorrelation) is not suf-
aration, because it allows to dene algorithms
with uniform performance i.e. whose behavior is
completely independent of the particular mixture.
Source distribution. The probability distribution
of each source is a `nuisance parameter': we are not
primarily interested in it, even though knowing
or estimating these distributions is necessary to
eciently estimate the parameter of interest. Even
if we say nothing about the distribution of each
source, we say a lot about their joint distribution
by assuming mutual source independence. This is
the key assumption. Source separation techniques
dier widely by the assumptions explicitly or im-
plicitly made on the individual distributions of
the sources. There is a whole range of possible
assumptions about the source distributions: they
cient to guarantee signal separation, it can be
put to use. Assume that the source signals have
unit variance so that Essy = I ; vector s is said
to be spatially white. Let W denote a `whitening
matrix' for x, that is z , W x is spatially white.
The composite transform WA necessarily is a ro-
tation matrix because it relates two spatially white
vectors s and z = WAs. Therefore, `whitening' or
`sphering' the data reduces the mixture to a ro-
tation matrix, leaving n(n 1)=2 unknown (rota-
tion) parameters to be determined by other than
second order information among the n2 unknown
parameters in A.
The prewhitening approach is sensible from an
algorithmic point of view but it is not statistically
ecient when the source distributions deviate sig-
are known in advance; some of their features (mo- nicantly from normality (see sec. 5.3). Actually,
ments,. . . ) are known; they belong to a parametric enforcing the whiteness constraint amounts to
family; no distribution model is available,. . . To believe that second order statistics are innitely
Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp. 1837-184

more reliable than any other kind of statistics.
This is, of course, untrue.
3. CONTRAST FUNCTIONS
Minimum contrast estimation is a general tech-
nique of statistical inference (Pfanzagl, 1973). It is
This shows that the maximum likelihood principle
is associated with a contrast function
ML [y] = K[yjs] (6)
and the normalized log-likelihood can be seen,
via (5) as an estimate of K[yjs]. The ML princi-
ple thus says something very simple when applied
relevant for blind deconvolution (see the inspiring
paper of Donoho (1981)) and has been introduced
in the related BSS problem in (Comon, 1994)
(even though in a somewhat restricted setting).
In both instances, a contrast function is a real
function of the probability distribution. To deal
with such functions, a special notation will be
useful: for x a given random variable, f (x) gener-
to the BSS problem: `nd matrix A such that the
distribution of A 1 x is as close as possible to the
hypothesized distribution of the sources'. The con-
trast function (6) is also arrived at via the infomax
principle (see (Bell and Sejnowski, 1995; Nadal
and Parga, 1994), and references therein).
The simple likelihood approach described above is
ically denotes a function of x while f [x] denotes a
function of the distribution of x. For instance, the
mean of x is the function m[x] , Ex.
Contrast functions for source separation are gener-
based on a xed hypothesis about the distribution
of the sources. A more powerful approach is to
model the data by adjusting both the unknown
system and the distributions of the sources. In
other words, one should minimize the divergence
ically denoted [y] and, by denition, verify
[C s]  [s] with equality only if y = C s is a
copy of the source signals. In other words: mix-
ing necessarily increases a contrast function and
separation is achieved by minimizing a contrast
function. Since the mixture can be reduced to a
rotation matrix by enforcing the whiteness con-
straint Eyyy = I (sect. 2.2), one can also consider
`orthogonal contrast functions': these are denoted
 [y] and must be minimized under the whiteness
constraint Eyyy = I .
3.1 Information theoretic contrasts
The maximum likelihood (ML) principle leads to
several contrasts which are expressed via the Kull-
back divergence. The Kullback divergence between
two probability density functions f (s) and g(s) on
K[yjs] with1 respect to A (via the distribution
of y = A x) and with respect to the model
distribution of s. This minimization problem has
a simple and intuitive theoretical solution. Denote
y~ a random vector with i) independent entries and
ii) each entry distributed as the corresponding
entry of y. A classic property (see e.g. (Cover and
Thomas, 1991)) of y~ is that
K[yjs] = K[yjy~ ] + K[~yjs] (7)
for any vector s with independent entries. Eq. (7)
shows that K[yjs] is minimized in s by simply tak-
ing s = y~ . Hence, the contrast function associated
to the `reduced likelihood':
MI [y] , K[yjy~ ] = min K[yjs]: (8)
s
Rn is dened as This is traditionally known as the mutual in-
Z 
f (s )

formation (between the entries of y). It veries
K(f jg) , f (s) log g(s) ds (3) MI [y]  0 with equality if and only if y is
whenever the integral exists. The divergence be-
tween the distributions of two random vectors w
and z is concisely denoted K[wjz].
If q(
) is an hypothesized pdf for the source vector
s, then the pdf of x = As is
p(x; A) = j det Aj 1 q(A 1 x) (4)
If T samples X1:T , [x(1); : : : ; x(T )] of x are mod-
eled as independent, then p(X1:T ) = p(x(1)) 



 p(x(T )). Thus, the probability p(X1:T ) of the
samples is readily computed as a function of the
unknown parameter, namely matrix A. Denoting
s a random vector with distribution q, simple
calculus shows that
1 T !1
T log p(X1:T ; A) ! K[A xjs] + cst:(5)
1
distributed as y~ . By denition of y~ , this happens
when the entries of y are independent. In other
words, MI [y] is a measure of independence be-
tween the entries of y.
Orthogonal contrasts. If the mixing matrix has
been reduced to a rotation matrix by whitening,
contrast functions like ML or MI can still be
used but the latter takes an interesting alternative
form under the whiteness constraint Eyyy = I :
X
MI [y] = H[yi] + cst (9)
i
Thus, minimizing the mutual information between
the entries of y is equivalent to minimizing the
sum of the entropies of the entries of y. There
is a simple interpretation: mixing the entries of s
`tends' to increase their entropies; it seems natural
Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp. 1837-184

to nd separated source signals as those with that if [y] = 0, then 4 [y] is equal (up to a
minimum marginal entropies. constant additive term) to
n
X
4 [y] , 2 ki Ciiii [y] = Ef4 (y) (12)
3.2 High order approximations i=1
High order statistics can be used to dene con- where we have dened f4(y) , 2 ni=1 ki (yi4 3).
P
trast functions which are simple approximations This is a pleasant nding: this contrast function
to those derived from the ML approach. The being the expectation of a function of y, it is par-
cumulants of the elements of a given vector y ticularly simple to estimate by a sample average.
are denoted Cij [y] , Cum[yi ; yj ] and Cijkl [y] ,
Cum[yi ; yj ; yk ; yl ]. Since the source vector s has Comon (1994) obtains another orthogonal con-
independent entries, all its cross-cumulants van- trast by approximating MI
ish: X
ICA [y] = 2 [y ]
Cijkl (13)
Cij [s] = i2 ij Cijkl [s] = ki ijkl (10) ijkl6=iiii

where  is the Kronecker symbol and we have Gaeta and Lacoume (1990) arrive at the same
dened the variance i2 and the kurtosis ki of
the i-th source as the second and fourth order
`auto-cumulants' of si : i2 , Cii [s] = Es2i and
ki , Ciiii [s] = Es4i 3E2s2i . The likelihood
contrast ML [y] = K[yjs] measures the mismatch
between output distribution and a model source
contrast by approximating the likelihood by a
Gram-Charlier expansion. This contrast actually
is similar to MI in the sense that it involves only
terms measuring the (4th order) independence
between the entries of y.
Independence can also be tested on a smaller
distribution. Cruder measures can be dened from subset of cross-cumulants with:
the quadratic mismatch between the cumulants: X
X JADE [y] , 2 [y ]:
Cijkl (14)
2 [ y ] , (Cij [y] Cij [s])2 : ijkl6=ijkk
ij
4 [ y ] ,
X
(Cijkl [y] Cijkl [s])2 :
ijkl
Clearly 2 is not a contrast because 2 [y] = 0 ex-
presses only the decorrelation between the entries
of y. On the contrary, one can show that 4 [y] is
a contrast if all the sources have known non zero
kurtosis. Even though fourth order information
is sucient by itself to solve the BSS problem,
it is interesting to use 2 and 4 in conjunction
because they jointly provide an approximation to
the likelihood contrast: if the sources are `close to
normal' and y is close to s then
K[yjs]  24 [y] , 481 (122[y] + 4[y]) : (11)
Room is lacking to discuss the validity of this
approximation. The point however is not to deter-
mine how closely 24 [y] approximates K[yjs] but
rather to follow the suggestion that second and
The motivation for using this specic subset is
that JADE also is a joint diagonalization crite-
rion, meaning that it can be optimized by an e-
cient algorithm, similar to the Jacobi technique of
diagonalization (Cardoso and Souloumiac, 1993).
Simpler contrasts can be used if the kurtosis of the
sources are known. For instance, eq. (12) suggests,
for negative kurtosis, a very simple contrast:
n
X
m [y] = Eyi4 (15)
i=1
This is actually a valid contrast function if ki +
kj < 0 for all pairs of sources.
4. ESTIMATING FUNCTIONS
In sect. 3, separating matrices were characterized
as the minimizers of contrast functions. In this
fourth order information could be used jointly. section, we derive the nite sample version of this
principle and the corresponding algorithms.
Orthogonal contrasts. We consider cumulant-based
orthogonal contrasts. The orthogonal approach,
which forces decorrelation i.e. 2 [y] = 0, corre- 4.1 A specic gradient for transformation models
sponds to replacing the factor 12 in eq. (11) by an
innite weight (optimal weighting is considered An appropriate notion of gradient is needed to
in (Cardoso et al., 1996); see also sec. 4.4) or describe the rst order variation of a contrast
equivalently to minimizing 4 [y] under the white- function [y] under linear transforms of y. By
ness constraint 2 [y] = 0. Simple algebra shows denition, an innitesimal linear transform of y
Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp. 1837-184

is close to the identity transformation, i.e. is in non-linear version 'i (yi ) of the ith output. The
the form I + E where E is a `small' matrix. It diagonal terms i = j x the scales of the outputs.
transforms y into (I + E )y = y + E y. If  is Similarly, maximizing the likelihood under the
smooth enough, we denote r[y] the n  n matrix whiteness constraint corresponds to solving (20)
the relative (Cardoso and Laheld, 1996)or the
natural (Amari, 1996) gradient of  at [y] such
that
[y + E y] = [y] + hr[y] j Ei + o(kEk)(16)
where h
j
i is the Euclidean scalar
product between
matrices: hM jN i = trace MN y = nij=1 Mij Nij .
P a symmetric part which is yyy I and expresses
The gradient of the Kullback divergence between
the distribution of the output y and the hypoth-
esized distribution of the sources is
rML [y] , rK[yjs] = EH' (y) (17)
where the vector-to matrix mapping Hs core is
H'(y) , '(y)yy I (18)
with '(y) , ['1 (y1 ); : : : ; 'n (yn )]y and where the
with with H = H' dened as
H' (y) , yyy I + '(y)yy y'(y)y (21)
Note that H' is decomposed in two parts: a skew-
symmetric part which is just H' (y) H' (y)y and
the whiteness condition.
Dierentiating other orthogonal contrast func-
tions yields similar estimating equations. For
instance, the simple 4th-order contrasts (12)
and (15) yield estimating equations in the form (21)
with ' functions respectively given by
'i (yi ) = ki yi3 and 'i (yi ) = yi3 (22)
respectively. The derivative of a contrast function
does not necessarily take the form of the mean
so-called `score functions' are dened as
0
'i , (log qi )0 'i (
) = qqi ((

))
i
Hence, with hypothetic pdf qi for the i-th source
is associated a non-linear function governing the
associated estimating equation. Note that if s
is a zero-mean unit-variance Gaussian variable,
the associated score is '(s) = s. In this case,
the estimating equation reduces to a condition of
empirical whiteness.
4.2 Estimating equations
More generally, an estimating function H for the
BSS problem is a vector-to matrix mapping H :
Rn ! Rnn . It is associated to the estimating
equation
T
1X H (y(t)) = 0
T t=1
thus called because, H being matrix-valued, equa-
value of H (y) for some function H . However the
contrast functions ICA and JADE are `asymp-
(19) totically associated' to the same estimating equa-
tion as 4 . This means that solving (20) with this
estimating function or minimizing ICA , JADE
or 4 with the cumulants estimated from T sam-
ples yields estimates which are equivalent for large
enough T .
4.3 Gradient algorithms
Relative gradient descent. The steepest descent
technique of minimization consists in moving by a
small step in a direction opposite to the gradient.
To minimize a contrast [y] by following the
steepest relative gradient, y undergoes a small
change into (I + E )y with E = r[y] for
a `small' positive step size . Thus, one step
of a relative gradient descent can be formally
described as
(20) y (I r[y])y = y r[y] y: (23)
According to (16), the resulting variation of [y]
is   hr[y]jEi = hr[y]j r[y]i =
tion (20) species a priori as many constraints
as unknown parameters in the BSS problem, so
that an estimate of the mixture (or of its inverse)
can be determined by solving eq. (20). Estimat-
ing equations and estimating functions provide a
kr[y]k2 < 0. The formal description (23) is
readily turned into o-line or on-line algorithms
when [y] has, like ML , a relative gradient in the
form r[y] = EH (y). Multiplication of y by (I
r[y]) as in eq. (23) amounts to changing B into
unied view of many BSS techniques.
The simplest example of estimating equation
stems from the log-likelihood whose stationary
points verify eq. (20) is with y = A 1 x and
H = H' (eq. (18)). For i 6= j , the ij -th term
of matrix equation (18) expresses the empirical
decorrelation between the j th output yj and a
(I r[y])B . In particular a stochastic gradient
algorithm is obtained by deleting the expectation
operator in r[y] = EH (y). Therefore, an on-line
stochastic relative algorithm updates a separating
matrix Bt when a new sample x(t) is received
according to
Bt+1 = Bt t H (y(t))Bt (24)
Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp. 1837-184
where t is a sequence of adaptation steps.
The most remarkable feature of gradient algo-
rithms in the form (24) is their uniform perfor-
mance property. This is to be understood in the
sense that the trajectory of the global system
Ct , Bt A is
Ct+1 = Ct t H (C s(t))Ct (25)
which does not depend on A: the only eect of
the mixing matrix is to determine (together with
B0 ) the initial value C0 = B0 A of the global
system. This is a very nice property because
the performance in terms of source separation
depends on the global system Ct and not on the
particular values of Bt and A. This is true for any
estimating function H ; however uniformly good
performance can only be expected if function
H is correctly selected, for instance by deriving
it from a contrast function. On line algorithms
based on function H in the form (18) are described
in (Amari et al., 1995); those based on form (21)
are studied in detail in (Cardoso and Laheld,
1996). The uniform performance property has also
been obtained in (Cichocki et al., 1994).
4.4 Adapting to the sources
When the pdf of a given source is assumed to
be q(s), this assumption manifests itself in the
5. PERFORMANCE ISSUES
5.1 Equivariance and uniform performance
The performance of a given BSS technique de-
pends a priori on the distribution of the source
signals and on the mixing matrix: performance is
expected to be poor for nearly Gaussian signals
and for nearly singular mixtures. However, this is
not true, at least in the high SNR domain: in the
limit of noise free observations, the BSS problem is
`uniformly hard in the mixing matrix'. This means
that the achievable separation is independent of
the matrix A: poorly conditioned mixtures can be
recovered just as easily as if A = I (see below)
Therefore, it seems possible to design algorithms
having `uniform performance' (with respect to A).
This is a very desirable property since such algo-
rithms can be studied and tuned independently of
the particular mixture to be separated.
It is not dicult to show that BSS algorithms
based on the solution of estimating equations
or on the optimization of contrast functions are
equivariant. We have already stressed (sec. 4.3)
that adaptive algorithms can also be made equiv-
ariant. See (Cardoso, 1995) for a general pre-
sentation and (Cardoso and Amari, 1997) for an
extension to location-scale models.
5.2 Stability
Let B be a separating matrix which is a station-
estimating functions like H' and H' via the score ary point of the learning rule (24), i.e. veries
function ' dened by (19). If the source has a in EH (y) = 0 for a given estimating function H .
fact a pdf r(s), one should rather use the `true' The question is to decide whether or not it is also
score function = rr . Pham (1997) has pro- a stable stationary point. It can be answered for
0

posed an elegant approach to easily approximate both the symmetric form H' and for the asym-
by a linear combination: metric form H' and, in both cases, depends only
the following non-linear moments
L
X
' (s) , l fl (s) (26) i , E'0i (si ) Es2i E'i (si )si (28)
l=1
where the normalization of the sources (scales) is
of a xed set of basis functions. Surprisingly, the
set of coecients minimizing the mean square
error E(' (s) (s))2 between the true score and
its approximation can be found without knowing
explicitly : it is
' (s) = EF 0 (s)y EF (s)F (s)y 1 F (s): (27)


where F (s) , [f1(s); : : : ; fL(s)]y is the L  1
column vector of basis functions and F 0 (s) is the
column vector of their derivatives. This is a very
determined by the condition EH (s) = 0.
Leaving aside the issue of the stability with re-
spect to scale, the analysis yields a pairwise sta-
bility condition for the symmetric form (21)
(1 + i )(1 + j ) > 1 for 1  i < j  n (29)
while the stability condition for the asymmetric
form (18) is that 1+ i > 0 for 1  i  n and that
i + j > 0 for 1  i < j  n: (30)
signicant result because the expression of '
can be simply estimated by replacing in (27) the Therefore stability appears to depend on pair-
expectations by sample averages and the values of wise conditions. Note that the stability domain
s by the estimated source signals. is larger for the symmetric form (21): this is a
Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp. 1837-184
consequence of letting the second order informa-
tion (the whiteness constraint) do `half the job'. In
both cases, a sucient stability condition is i > 0
is
 , E 2 (s)Es2 E2[ (s)s] and
for all the sources. Another important point is
=
 actually is the best
that i = 0 for any 'i if the ith source is normally
distributed. Hence, the stability conditions can
never be met if there is more than one Gaussian
source.
We have considered linear-cubic score functions in
 , this moment characterizes the hard-
when ' = i.e. when the non-linear function
matches the true score function. The maximum
value of
expression (34) with
achievable ISR rate with T independent samples
(the asymptotic Cramer-Rao bound, see (Yellin
and Friedlander, 1996; Pham and Garrat, 1997)).
Finally, the achievable performance depending on
the size of
sect. 4. If 'i (si ) = i si + i s3i for two constants i ness of the BSS problem with respect to source
and i , then i = i (3E2 s2i Es4i ) = i ki . There-
fore, if one wishes to use cubic non-linearities, it is
sucient to know the sign of the kurtosis of each
source to make separating matrices stable. For
other than cubic scores, the sign of the kurtosis
is not relevant to stability.
distribution. It tends to zero when the common
source distribution tends to a normal distribution,
showing how the achievable performance worsens
when the source distributions tend to be normally
distributed. It tends to +1 when the source dis-
tributions tend to discrete or to bounded sup-
port. In the case of discrete sources, deterministic
(error-free) blind identication is possible with a
5.3 Accuracy of estimating equations nite number of samples. In the case of sources
with bounded support, the MSE of blind identi-
The interference-to-signal ratio (ISR) obtained in cation decreases at a much faster rate than the
rejecting the qth source in the estimate of the pth 1=T rate obtained for nite values of
.
source by a separating matrix B is
(BA)2 Es2
pq (B ) , (BA)2pq Esq2 p 6= q (31)
pp p 6. CONCLUSION
Let BbT be the separating matrix obtained via a In this short review, the focus was on some basic
particular algorithm using T samples. The limit principles underlying the simplest source separa-
tion model. Many interesting issues have been left
ISRpq , Tlim
!1
T E (BbT ) (32) out, like discussing the connections between BSS
and blind deconvolution; convergence speed of
usually exists and provides an asymptotic measure adaptive algorithms; design of consistent estima-
of performance of separation of a given o-line tors based on noisy observations, detection of the
BSS technique. If the procedure is equivariant, number of sources, etc. . . Rather than concluding
this measure does not depend on A. For simplicity, by a summary, we brie
y comment on three other
we consider identically distributed signals and issues.
identical non-linear functions: 'i (
) = '(
), so
that i =  and
i =
for 1  i  n where Algebraic approaches. The 4th order cumulants
of x have a very regular structure in the BSS

i , E'2i (si )Es2i E2 ['i (si )si ]  0: (33) model which calls calls for algebraic approaches.
Simple algorithms can be based on the eigen-
If the score function has been obtained by the structure of `cumulant matrices' built from cu-
technique of Pham described in sec. 4.4 based on mulants (Cardoso, 1989; Tong et al., 1993). An
T independent samples, then the rejection rates exciting direction of research is to investigate
ISRpq and ISRpq obtained with the estimating high-order decompositions that would generalize
function H' and H' respectively are (for p 6= q): matrix factorizations like SVD or EVD to 4th
 
order cumulants (Lathauwer and B. De Moor,
ISRpq = ISR = 21
1 +
+1 2 (34) 1996; Cardoso, 1991; Comon and Mourrain, 1994).
  Using temporal correlation. If the source signals
ISRpq = ISR = 21
1 + 21 (35) are temporally correlated, time structure can also
be exploited. It is possible to achieve separation
These results teach several things. First, ISR is if all the source signals have distinct spectra even
lower bounded by 1=4 regardless of the value of if each source signal is a Gaussian process (Tong

: this is a general property of orthogonal BSS et al., 1991b). Simple algebraic techniques can
techniques (Cardoso, 1994) and is the price to pay be devised (see (Tong et al., 1990; Belouchrani
for blindly trusting second order statistics i.e. for et al., 1997)); the Whittle approximation to the
whitening. Second, ISR and ISR are both mini- likelihood is investigated in (Pham and Garat,
mized by maximizing
. This is achieved precisely 1993).
Proceedings of the SYSID’97, 11th IFAC symposium on system identification. pp. 1837-184

Deterministic identication. As indicated in sec. 5.3, Comon, P. (1994). Independent component analy-
sources with discrete support allow for deter- sis, a new concept ?. Signal Processing, Else-
ministic identication (innite Fisher informa- vier 36(3), 287{314. Special issue on Higher-
tion). Specic contrast functions can be de- Order Statistics.
vised (Gamboa, 1995) to take advantage of dis- Comon, P. and B. Mourrain (1994). Decomposi-
creteness. There is a rich domain of application tion of quantics in sums of powers. In: SPIE
with digital communication signals where infor- conference on Advanced Signal Processing.
mation is coded with discrete symbols. San Diego.
Cover, Thomas M. and Joy A. Thomas (1991).
Elements of information theory. Wiley series
7. REFERENCES
in telecommunications. John Wiley.
Amari, S. (1996). Neural learning in structured Donoho, D. (1981). On minimum entropy decon-
parameter spaces|natural Riemannian gra- volution. In: Applied time-series analysis II.
dient. In: Proc. NIPS. pp. 565{609. Academic Press.
Amari, S.-I., A. Cichocki and H.H. Yang (1995). Gaeta, Michel and Jean-Louis Lacoume (1990).
Recurrent neural networks for blind separa- Source separation without a priori knowledge:
tion of sources,. In: Proc. Int. Symp. NOLTA. the maximum likelihood solution. In: Proc.
pp. 37{42. EUSIPCO . pp. 621{624.
Bell, A. J. and T. J. Sejnowski (1995). An Gamboa, Fabrice (1995). Separation of sources
information-maximisation approach to blind having unknown discrete supports. In: Proc.
separation and blind deconvolution. Neural IEEE SP Workshop on Higher-Order Stat.,
computation 7(6), 1004{1034. Aiguablava, Spain. pp. 56{59.
Belouchrani, Adel, Karim Abed Meraim, Jean- Lathauwer, L. De and J. Vandewalle B. De Moor

Francois Cardoso and Eric Moulines (1997). (1996). Independent component analysis
A blind source separation technique based on based on higher-order statistics only. In: Proc.
second order statistics. IEEE Trans. on S.P. IEEE SSAP workshop, Corfu . pp. 356{359.
45(2), 434{44. Nadal, J.-P. and N. Parga (1994). Nonlinear neu-
Cardoso, Jean-Francois (1989). Source separa- rons in the low-noise limit: a factorial code
tion using higher order moments. In: Proc. maximizes information transfer. NETWORK
ICASSP. pp. 2109{2112. 5 , 565{581.
Cardoso, Jean-Francois (1991). Super-symmetric Pfanzagl, J. (1973). Asymptotic expansions re-
decomposition of the fourth-order cumulant lated to minimum contrast estimators. The
tensor. Blind identication of more sources Annals of Statistics 1(6), 993{1026.
than sensors. In: Proc. ICASSP. pp. 3109{ Pham, Dinh-Tuan and Philippe Garrat (1997).
3112. Blind separation of mixture of independent
Cardoso, Jean-Francois (1994). On the perfor- sources through a quasi-maximum likelihood
mance of source separation algorithms. In: approach. IEEE Tr. SP. To appear.
Proc. EUSIPCO. Edinburgh. pp. 776{779. Pham, D.T. and P. Garat (1993). Separation
Cardoso, Jean-Francois (1995). The equivariant aveugle de sources temporellement correlees.
approach to source separation. In: Proc. In: Proc. GRETSI . pp. 317{320.
NOLTA. pp. 55{60. Tong, L., R. Liu, V.C. Soon and Y. Huang
Cardoso, Jean-Francois and Antoine Souloumiac (1991a). Indeterminacy and identiability
(1993). Blind beamforming for non Gaussian of blind identication. IEEE Tr. on CS
signals. IEE Proceedings-F 140(6), 362{370. 38(5), 499{509.
Cardoso, Jean-Francois and Beate Laheld (1996). Tong, L., V.C. Soon, Y.F. Huang and R. Liu
Equivariant adaptive source separation. (1991 b). A necessary and sucient condition
IEEE Trans. on S.P. 44(12), 3017{3030. for the blind identication of memoryless sys-
Cardoso, Jean-Francois and Shun-Ichi Amari tems. In: Proc. ISCAS. Vol. 1. Singapore.
(1997). Maximum likelihood source separa- pp. 1{4.
tion: equivariance and adaptivity. In: To ap- Tong, L., Y.F. Huang V.C. Soon and R. Liu
pear in Proc. of SYSID'97.
Cardoso, Jean-Francois, Sandip Bose and Ben-
jamin Friedlander (1996). On optimal source
separation based on second and fourth or-
der cumulants. In: Proc. IEEE Workshop on
SSAP, Corfou, Greece.
Cichocki, A., R. Unbehauen and E. Rum-
(1990). AMUSE: a new blind identication
algorithm. In: Proc. ISCAS.
Tong, Lang, Yujiro Inouye and Ruey-wen Liu
(1993). Waveform preserving blind estimation
of multiple independent sources. IEEE Tr. on
SP 41(7), 2461{2470.
Yellin, D. and B. Friedlander (1996). Multi-
mert (1994). Robust learning algorithm for channel system identication and deconvo-
blind separation of signals. Electronic letters lution: performance bounds. In: Proc. IEEE
30(17), 1386{87. SSAP workshop, Corfu. pp. 582{585.