5 views

Original Title: CRB for Low-Rank Decomposition

Uploaded by Uma Mageshwari

CRB for Low-Rank Decomposition

rank

© All Rights Reserved

- Chapter 09
- Partial Least Squares Regression a Tutorial
- Matrices
- 2nd PU Maths Model QP 2
- UT Dallas Syllabus for engr2300.002.11f taught by Naofal Al-Dhahir (nxa028000)
- Ch 4 Determinant
- WAVELET_PSO-BASED_LQR_ALGORITHM_FOR_OPTI.pdf
- AT-2016-17-C-XI _PAPER-1_-AT+PCM.pdf
- ANALISIS SEUDOTRIDIMENSIONAL
- RPT Matematik T5
- lucerogaytan magic number
- SM1421701_Sol
- nmf-mod
- hw1
- 12th First Term Paper Maths 2011-12-1
- mazzia
- Space-Time Behavior-based Correlation-Or-How to Tell if Two Underlying Motion Fields Are Similar Without Computing Them - Shechtman Irani - IEEE Transactions on Pattern Analysis and Machine Intelligence - 2007
- assessmnet 02
- Book Linear34546456
- Ans 1 Smooth

You are on page 1of 13

9, SEPTEMBER 2001

Decomposition of Multidimensional Arrays

Xiangqian Liu and Nicholas D. Sidiropoulos, Senior Member, IEEE

AbstractUnlike low-rank matrix decomposition, which is fact was illustrated by Cattell and rigorously established under

generically nonunique for rank greater than one, low-rank three- increasingly milder conditions over a span of some 30 years by

and higher dimensional array decomposition is unique, provided

various authors [8][10], [14], [15], [20], [21], [25].

that the array rank is lower than a certain bound, and the correct

number of components (equal to array rank) is sought in the Low-rank decomposition of multidimensional arrays has

decomposition. Parallel factor (PARAFAC) analysis is a common been used as as a data analysis tool in an unusual variety

name for low-rank decomposition of higher dimensional arrays. of disciplines, e.g., [2], [5], [6], [10], [14], and [16], to

This paper develops CramrRao Bound (CRB) results for mention a few. More recently, it was shown to have exten-

low-rank decomposition of three- and four-dimensional (3-D and

4-D) arrays, illustrates the behavior of the resulting bounds, and sive application in signal processing and communications,

compares alternating least squares algorithms that are commonly including blind multiuser detection in direct-sequence code-di-

used to compute such decompositions with the respective CRBs. vision multiple-access (DS-CDMA) communications [25],

Simple-to-check necessary conditions for a unique low-rank multiple-invariance sensor array processing [24], blind beam-

decomposition are also provided.

forming in specular multipath [17], [26], and, in more general

Index TermsCramrRao bound, least squares method, ma- terms, blind diversity-combining in communications [22]. De-

trix decomposition, multidimensional signal processing. spite these wide-ranging applications, pertinent CramrRao

bounds (CRBs) for low-rank decomposition of multidimen-

I. INTRODUCTION sional arrays have been missing from the literature. The first

contribution of this paper is the development of CRBs for

rays], the low-rank property in itself is not enough to

guarantee a unique data model, and one has to resort to

low-rank decomposition of 3-D and four-dimensional (4-D)

arrays. Complex model parameters and a complex circularly

symmetric white Gaussian noise model are assumed, but some

additional problem-specific structural properties to obtain a

special cases are also considered due to their importance in

unique parameterization. Examples include orthogonality (as

applications: a 3-D array with Vandermonde structure in one

in singular value decomposition), Vandermonde, Toeplitz,

dimension, and the real-parameter 3-D case.

or finite-alphabet constraints. Notwithstanding the lack of

A wide variety of algorithms have been developed for com-

inherent uniqueness, low-rank matrix decomposition plays a

puting low-rank decomposition in three and higher dimensions.

key role in modern signal processing.

These range from eigenvalue-type algebraic techniques [19],

The concept of rank can be extended to arrays in three or

[20] to alternating least squares (ALS) algorithms [8][10]; see

higher dimensions in a natural way [14], [15]. Low-rank decom-

[25] for a readily available reference and [5] and [23] for a tuto-

position of three-dimensional (3-D) arrays was developed by

rial overview. Among these possibilities, ALS is the workhorse

Harshman [8][10] under the name parallel factor (PARAFAC)

technique that is mostly preferred in practice [5], [23]. Even

analysis and independently by Carroll and Chang [6] under the

though ALS has been used extensively in this context and it has

name canonical decomposition (CANDECOMP), building on a

many desirable properties, its performance has not previously

inconspicuous principle proposed in the factor analysis litera-

been measured relative to the CRB. This is the second contribu-

ture by Cattell [7]. Interestingly, these developments actually

tion of this paper: putting ALS to the test versus the CRB and

preceded the fundamental work on higher dimensional array

verifying that, indeed, the performance of ALS comes close to

rank [14], [15]. Quite unlike low-rank matrix decomposition,

the CRB. This means that the CRB can be used to predict the av-

which is generically nonunique for any rank greater than one,

erage behavior of ALS, which in turn makes the CRB a valuable

low-rank, 3-D array decomposition is essentially unique for a

design tool. For example, in the context of DS-CDMA com-

meaningful range of low-enough ranks [14], and the situation

munications [25] and diversity-combining [22] in more gen-

actually improves in higher dimensions [21]. This remarkable

eral terms, the CRB allows quantitative evaluation of the di-

versity tradeoff for a specified performance. The real-parameter

Manuscript received July 17, 2000; revised May 31, 2001. This work was Gaussian CRB for low-rank decomposition of 3-D data can be

supported by the National Science Foundation under CAREER grant 0096165

and Wireless grant 0096164. The associate editor coordinating the review of this used to improve experimental design in PARAFAC applications

paper and approving it for publication was Dr. Brian Sadler. in chemistry; see [5] and references therein.

The authors are with the Department of Electrical and Computer Engi- The paper is organized as follows. The model is introduced

neering, University of Minnesota, Minneapolis, MN 55455 USA (e-mail:

xiang@ece.umn.edu; nikos@ece.umn.edu). in Section II, including sufficient uniqueness conditions.

Publisher Item Identifier S 1053-587X(01)07057-X. Section III provides a starting point and roadmap for the

1053587X/01$10.00 2001 IEEE

LIU AND SIDIROPOULOS: CRAMRRAO LOWER BOUNDS FOR LOW-RANK DECOMPOSITION 2075

pertinent CRBs (associated derivations are deferred to the minimum number of rank-one (three-way) components needed

Appendices). These bounds are discussed and illustrated in to decompose .

Section IV. Simple-to-check necessary uniqueness conditions Define an matrix with typical element

are developed in Section V. Multilinear ALS is compared with , matrix with , matrix

the CRB in Section VI. Conclusions are drawn in Section VII. with , and matrices with

Some notation conventions that will be used in this paper . Compact matrix representations of the model in (3) are

follow. made possible by employing the KhatriRao matrix product.

Transpose of . For example

Complex conjugate of .

Conjugate transpose of .

th column of . .. ..

th column of ( is the th row of ). . .

Kronecker product of and .

(4)

KhatriRao (column-wise Kronecker) product of

The superscript means that the matrix is of size

and .

and that the -index ( goes first in the product ) runs faster

the Hadamard (element-wise) product of and :

than the -index along its rows. By symmetry

;

Diagonal matrix constructed out of the th row of (5)

.

Kronecker delta: when , and and

when , and are integers.

(6)

Frobenius norm.

Matrix pseudo-inverse. Next, consider an 4-D array with typical

element

II. MULTIDIMENSIONAL LOW-RANK MODELING

Consider an matrix . rank if and only if (7)

can be written as a sum of but no less than rank-one

matrices (vector outer products) for , , , and .

Define a matrix with typical element

(1)

and matrix with . Similar to the

where is , and is . Note that we may assume three-way case, the KhatriRao product can be used to cast the

without loss of generality that the columns of and are lin- model in (7) in matrix form. For example

early independent; otherwise, at least one of the rank-one

(8)

components can be absorbed in the others. In general,

, but if , then constitutes a The KhatriRao product has the property

low-rank decomposition of . Let denote the th entry [13]; in addition, the order of KhatriRao multi-

of . A scalar view of the relationship is plications only affects the order of rows in the final result [13].

In particular, rank is not affected by the order in which the mul-

(2) tiplications are carried out.

Further generalizing to dimensions

Observe that is written as a sum of double products, i.e., it (9)

admits an -component bilinear decomposition.

Next, consider an 3-D (also known as three-way)

array with typical element , and the component tri- for , , where . Upon

linear decomposition: defining matrices with , many

matrix representations are possible, e.g.,

(3)

(3) expresses the three-way array as a sum of rank-one A. Uniqueness

three-way factors, each one of which is the outer product of Definition 1: The -rank of a matrix (which is

three vectors. Analogous to the definition of matrix (two-way denoted by ) is if and only if every columns of are

array) rank, the rank of a three-way array is defined as the linearly independent and either has columns or contains

2076 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 9, SEPTEMBER 2001

a set of linearly dependent columns. Note that -rank is which certain simplifications are possible due to the fact that

always less than or equal to rank the noise covariance matrix can be written in convenient closed

, . -rank stands for Kruskal-rank [14]; the term form and an alternative scalar computation approach can be

was coined by Harshman and Lundy [11]. adopted. The real-parameter three-way case is the one that is

Theorem 1: (Sufficient condition for uniqueness of low-rank mostly encountered in applications of PARAFAC in other disci-

decomposition of -way arrays [21], [22]; cf. [14] for a basic plines. Under the Gaussian assumption, the CRB will be block

precursor result, and [25] for its complex counterpart). Consider diagonal in the noise and signal parameters. In the interest of

the -component -way array given in (9), and suppose that it brevity, we therefore assume that the noise variance is known

is irreducible (i.e., the rank of the -way array is ). If throughout the derivations.

A delicate point regarding these CRBs is the inherent per-

mutation and scale ambiguity. For example, in the three-way

(10) case (3), one can clearly reshuffle the components and/or set

, , and such that

then the -way array , , , , and remains unaffected. In other

has unique rank-one -way factors words

, , such that . In fact,

Note that the three-way array rank must be low enough low-rank decomposition of multidimensional arrays is unique

relative to the maximum possible sum of -ranks, e.g., (under the sum of -ranks condition) up to the above indetermi-

in the nacy, which is unresolvable but also mostly insignificant. Some

three-way case,1 and the rank of the decomposition (number of means of fixing permutation and scale should be agreed upon in

columns of ) must be equal to the correct rank . If the order to obtain a meaningful bound. A simple (but not the only)

latter is less than , then exact decomposition is impossible, way of doing this in the three-way case is to fix the first row of

by definition of array rank. If the rank of the decomposition is and to (this takes care of scale ambiguity) and

greater than , then the decomposition is not unique. In addi- further assume that the first row of is known and consists of

tion, note that increasing the number of dimensions decreases distinct elements (which subsequently resolves the permutation

the -rank requirement per dimension. In three dimensions, ambiguity).2

it is necessary for uniqueness that the -rank of any matrix is Our convention is plausible in the context of antenna array

at least two. In four dimensions and beyond, even this is not reception of DS-CDMA. Let

necessary: It is possible to have one matrix of -rank equal to antenna array response (steering) matrix;

one and still have uniqueness. Regarding conditions that are code matrix;

necessary for uniqueness, see also Theorem 2 in Section V. symbol matrix.

Since the first element of the antenna array simply serves as an

III. ROADMAP OF CRB RESULTS arbitrary frame of reference, the first row of can be assumed

As mentioned before, although low-rank decomposition of to be a row vector of all ones, without loss of generality. Sim-

multidimensional arrays has been used as a data analysis tool in ilarly, the first row of can be assumed to be a row vector of

a variety of disciplines, pertinent CRB results have been missing all ones (first chip of all users equal to one) without loss of gen-

from the literature. The parameters of interest for which the erality. Then, the scale factor (propagation loss and phase shift

CRB will be established are the elements of the unknown ma- times reference antenna gain times first chip) for each user can

trices that are involved in the decomposition ( in the be absorbed in the corresponding column of the symbol ma-

three-way case). trix . It then seems reasonable to assume that the first row

In the Appendixes, we derive CRBs for low-rank decomposi- of consists of distinct elements, although considering it to

tion of complex-parameter three-way arrays (Appendix A) and be known is less appealing in a blind setting. Note, however,

complex-parameter four-way arrays (Appendix B), both in i.i.d. that this is just a way of technically fixing an inherently unre-

complex circularly symmetric Gaussian noise. These are ap- solvable ambiguity for the purpose of performance evaluation.

plicable, e.g., in diversity-combining applications in communi- Permutation and scale-fixing conventions could be application

cations [22], [25]. In Appendix C, we consider the case of a dependent and, indeed, could influence the final bound. What

complex-parameter three-way array that exhibits Vandermonde is less restrictive in one application could be more restrictive in

structure in one dimension. This is of interest in sensor array anotherwe do not have a universal solution, but one could still

processing applications [24], and it can be viewed as gener- use the results in the Appendixes to get a head start in computing

alizing corresponding results for the two-way case with Van- the pertinent bounds under alternative conventions.

dermonde structure in one dimension [27]. In Appendix D, we 2It is not enough to assume that the first row of A is known. The reason is

consider the special case of real-parameter three-way arrays for that scale ambiguity has to be fixed prior to resolving the permutation ambiguity;

without knowing which column is which, we cannot divide by the first element

1Three-way array rank could be up to min(I J; J K; I K ); see [15]. because we do not know what this element is.

LIU AND SIDIROPOULOS: CRAMRRAO LOWER BOUNDS FOR LOW-RANK DECOMPOSITION 2077

Fig. 1. PARAFAC CRB: Varying sample size along one dimension (number

C

of rows of ).

In this section, we illustrate the behavior of the CRBs derived

in the Appendixes. Throughout, the CRB is first normalized in

an element-wise fashion, i.e., each unknown parameters CRB is

divided by the corresponding parameters modulus square. The

average CRB of all the unknown parameters is then used as a

single performance measure. Therefore, instead of plotting the

trace of the CRB, we plot a weighted trace, with weights propor-

tional to the inverse modulus square of the respective parame-

ters. This measures average variance relative to modulus square

across all parameters of interest. Signal-to-noise ratio (SNR) is

defined as [cf. (4) and (8)]

SNR (three-way array) Fig. 3. PARAFAC CRB: With versus without assuming Vandermonde

structure.

SNR (four-way array)

component matrices , , and is compared with the corre-

Unless otherwise specified, the matrices , , and were ran- sponding bound for a four-way array (Appendix B) obtained by

domly drawn from an i.i.d. standard Gaussian distribution, and augmenting the three-way model by adding another dimension

the CRB is averaged over all the unknown parameters of all ma- of size , , or . We have seen that higher dimen-

trices. sionality leads to a relaxed uniqueness condition (Theorem 1 ).

Fig. 1 is an illustration of the behavior of the CRB for Fig. 2 demonstrates that higher dimensionality also benefits in

low-rank decomposition of a complex-parameter three-way terms of CRB.

array (Appendix A) as the size of one dimension increases, with In Fig. 3, we compare the CRB of Appendix A with that of

SNR fixed at 10 dB. In this experiment, and are , Appendix C for a given three-way array with Vandermonde

whereas is augmented from to . Notice that structure in one dimension. The plot shows average CRB of

the number of parameters also increases with the sample size. the generators of the Vandermonde matrix. The bound in Ap-

However, the number of equations (data points) increases faster pendix A ignores the fact that Vandermonde structure is present

than the number of unknown parameters ( versus order of and is therefore above the bound in Appendix C. Comparisons

, respectively). One would then intuitively expect of this kind can help gauge the performance margin that can

that the CRB should decrease with increasing sample size along be gained by employing and exploiting additional model

any dimension. Although this seems to be the trend, it is not structurein this case, Vandermonde. An example could be

true in general. We have seen examples of datasets where the the choice of complex exponential (OFDMA-like) spreading

average CRB may actually increase before decreasing again as codes in the context of [25].

one adds rows to . A quadrilinear model can always be viewed as a trilinear

Fig. 2 illustrates the effect of adding an extra diversity dimen- model, i.e., in (8), let to obtain

sion. The CRB for a three-way array (Appendix A) with . This unfolding into a lower dimensional model

2078 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 9, SEPTEMBER 2001

Fig. 4. PARAFAC CRB: Model viewed as trilinear versus quadrilinear. Fig. 5. PARAFAC CRB: CDMA diversity-combining tradeoff.

sional model is uniquely parameterized, the loss of structure

will result in a loss of statistical efficiency. This is illustrated

in Fig. 4, which plots the CRB in Appendix B for a quadrilinear

data model with , , , and all and the CRB in Ap-

pendix A corresponding to unfolding it into a trilinear model.

Notice that the difference is significant.

The bounds in Appendices AD can be used to gauge at

the diversity-combining tradeoff for system design purposes.

As an example, consider a DS-CDMA system with

users, spread using WalshHadamard codes of length

chips/symbol, and received at the base station using a ULA con-

sisting of antennas spaced half a wavelength apart. Col-

lect symbols worth of samples from the baseband out-

puts of the antenna elements sampled at the chip rate. As shown

in [25], the resulting data can be arranged into an Fig. 6. PARAFAC CRB: performance differential when one matrix is known.

three-way array of rank , and hence, the steering vectors in

, spreading codes in , and transmitted symbols in can formance of estimation algorithms that treat as a deterministic

all be blindly recovered. Now, suppose that one wishes to im- unknown.

prove blind estimation performance and that two options are In some instances, the signals along one of the diversity di-

available: increase spreading, [going from to mensions (one of the three matrices , , and ) may be

chips/symbol (at the expense of doubling the bandwidth)] or known. Examples include known spreading codes or estimated

double the number of receive antennas in the ULA from to signatures or training signals. One may then wonder how this

(paying the cost of extra hardware). Aside from other con- knowledge affects the estimation of the remaining unknowns.

siderations, which of the two options is best performance-wise? Interestingly, the answer is often not much. Fig. 6 depicts the

Using the CRB in Appendix C (notice that the steering vectors in average CRB for the free elements of and when is as-

are Vandermonde, due to the ULA assumption), Fig. 5 shows sumed known versus unknown. All three matrices are , and

that for directions 20 , 40 , 60 , and 80 relative to the array the CRB curves are averaged over 1000 realizations of randomly

broadside, it is best to double the number of antennas, rather drawn , , and . The curves come closer for higher , , and

than the spreading gain. Note that the CRB in Fig. 5 is averaged relative to . We conclude that there is often enough struc-

over 100 random realizations of the symbol matrix , and the ture in the trilinear model itself so that knowledge of the signals

situation reverses for closely spaced directions, as expected. The in one dimension does not significantly improve the estimation

reason we average the CRB over random realizations of is as of signals in the other dimensions.

follows. Let denote the parameter vector to be estimated. Any In general, uniqueness neither implies nor is implied by a

estimation algorithm that treats as a deterministic unknown finite CRB. A simple example in [1] can be used to clarify this.

is bounded by CRB pointwise in . Hence, the average per- Consider the estimation of parameter from the measurement

formance of any such algorithm over a collection of is , where is a zero-mean, Gaussian random variable

bounded by the average of the respective CRBs over . For a of variance . The Fisher information is proportional to ,

large number of representative drawn via Monte Carlo simu- which is 0 at , although is the only identifiable

lation, such averages can be used to gauge the achievable per- value of . Hence, uniqueness does not imply a finite CRB. For

LIU AND SIDIROPOULOS: CRAMRRAO LOWER BOUNDS FOR LOW-RANK DECOMPOSITION 2079

It follows that is not unique. Similarly, from (5) and (6), we

conclude that if or , then or cannot

be unique. For uniqueness of the model in (9), all three condi-

tions need to hold; hence, condition (11) follows. For the

case, consider (8). There are a total of four leave-one-out selec-

tions from , and the KhatriRao product of the

three selected matrices must be full rank for the left-out matrix

to be unique (recall that KhatriRao product rank is not affected

by the order in which the multiplications are carried out).

Remark 1: In order to see that the sufficient condition (10)

in Theorem 1 implies the necessary condition (11) in Theorem

Fig. 7. CRB behavior near nonidentifiability. 2, consider (10) with , and note that for any ,

it holds that ; hence, (10) implies

any , the Fisher information is nonzero, and hence, the

(12)

CRB is finite, whereas these are not identifiable. Therefore,

uniqueness is not implied by a finite CRB. However, it is natural which in turn implies

to expect that as one moves closer to nonuniqueness, the CRB

should increase. This is the subject of Fig. 7. Let and be (13)

matrices with full rank, and let

It has been shown in [26] that ;

thus, (13) yields , which implies . The

remaining two implications follow by symmetry of (9) and (10).

Remark 2: Testing whether the necessary condition is

valid is much simpler than checking the sufficient condition,

where . When , , and which involves -rank rather than rank. Checking the -rank

, the model is unique according to (10). implies of a matrix involves sequentially checking all possible selec-

, and hence, the sufficient condition (10) is violated: tions of columns ( going from to 2 or vice-versa) for

. Fig. 7 depicts the behavior of the CRB linear independence. In the worst case, the calculation of ,

as approaches zero. Observe that the CRB remains stable for , and by singular value decomposition requires up to

a wide range of values and then increases sharply in the vicinity floating-point operations

of . This is consistent with practical experience involving (flops), whereas the calculation of , , and

ALS applied to real data sets exhibiting near-collinearity [5]. takes only flops.

The KhatriRao product can be viewed as a selection of

V. NECESSARY UNIQUENESS CONDITIONS columns from the Kronecker product, whose rank is the product

of ranks of its constituent matrix factors. Hence

Condition (10) is sufficient but not necessary for uniqueness.

When low-rank modeling is utilized as an exploratory design

tool, it is useful to have simple tests that quickly rule out non-

identifiable models. This is the subject of this section. Therefore, requires , and we obtain the

Theorem 2: For , a necessary condition for unique- following further simplified condition.

ness of the model in (9) is Corollary 1: For , a further simplified necessary con-

dition for uniqueness of the model in (9) is

(11) (14)

For , a necessary condition is

dimensions.

The principle of ALS can be used to fit low-rank models in

The result is further generalized for any : The KhatriRao any dimension. A trilinear ALS (TALS) algorithm can be found

product of any leave-one-out selection of matrices from in [25], but the idea of using ALS to fit low-rank three-way

must be of rank for the model in (9) to be models goes back to Harshman [8][10]. The basic idea be-

unique. hind ALS is simple: each time update one matrix, using least

2080 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 9, SEPTEMBER 2001

Fig. 8. TALS performance vesus CRB. (a) A. (b) B. (c) C. (d) Average. Fig. 9. QALS performance versus CRB. (a) A. (b) B. (c) G. (d) H.

squares (LS) conditioned on previously obtained estimates for

the remaining matrices; proceed to update the other matrices;

repeat until convergence of the LS cost function (this is guaran-

teed for ALS algorithms, e.g., [25]). For example, quadrilinear

ALS (QALS) aims to

symmetry of the quadrilinear model [cf. (7)], the conditional

least square updates are

model parameters versus trace(CRB) . The

remaining three plots present average MSE for the free elements

where , , , and denote running estimates of , , , of , , and respectively, versus the corresponding bound.

and . Figs. 9 and 10 depict Monte Carlo simulation results com-

For zero-mean white (in all dimensions) Gaussian noise, ALS paring QALS performance to the CRB for a typical scenario. For

yields maximum likelihood (ML) estimates, provided the global this simulation, , , and 800 Monte

minimum is reached. Under mild regularity conditions, ML is Carlo trials per datum have been conducted. Fig. 9 plots the es-

asymptotically (in sample size) unbiased and asymptotically timation MSE for the free elements of , , , and , respec-

achieves the CRB [12, Ch. 7]. For signal-in-noise problems, tively, versus the corresponding bound. Fig. 10 presents average

ML also achieves the CRB for high-enough signal-to-noise MSE for all free model parameters versus the CRB. Observe

ratios [12, Ch. 7]. It therefore makes sense to compare trilinear that both TALS and QALS remain close to the CRB for a wide

and quadrilinear ALS against the respective CRBs. range of SNRs and reasonably small sample sizes in all dimen-

In all of our Monte Carlo results, ALS is randomly initialized sionsbut note that total sample size is the product of sample

once per trial and then iterated until convergence. No reinitial- sizes along the individual dimensions; hence, we have 8000

izations are used. samples for the trilinear example in Fig. 8 and 1296 samples for

Fig. 8 depicts simulation results comparing TALS perfor- the more structured quadrilinear model in Figs. 9 and 10. These

mance to the CRB for a unconstrained trilinear model. In this are typical sample sizes in applications, wherein the number

simulation, , , and 200 Monte of samples in one dimension may be related to spreading gain,

Carlo trials per datum have been conducted. The lower right number of symbols collected, etc. [5], [25]. Total sample size is

plot presents average mean squared error (MSE) for all free what matters for the good asymptotic properties of ML to come

LIU AND SIDIROPOULOS: CRAMRRAO LOWER BOUNDS FOR LOW-RANK DECOMPOSITION 2081

into play, and this explains the good behavior of ALS, which where , , and are the corresponding noise matrices.

aims for the ML solution. Observe that the gap between ALS Therefore, we can write the likelihood function of in three

and the CRB reduces quickly with increasing SNR (note that equivalent ways:

the plots are in log-log scale). The reasons for the nonzero gap

at finite SNR are as follows.

Practical experience [5] with trilinear ALS shows that the

global minimum is usually reached as long as the con-

stituent rank-one factors are not essentially collinear along

any dimension. Nevertheless, for finite SNR, there exist a

few inevitable bad runs, which hit a local minimum.

Even if the right minimum is reached, scale and, more im-

portantly, permutation matching may fail due to residual

noise. This effect is particularly pronounced at low SNR.

In practice, some prespecified tolerance threshold is used

where denotes the th column of and similarly for and

to terminate the ALS iteration. In certain situations, con-

.

vergence can be relatively slow, which may lead to prema-

As explained in Section III, a delicate point regarding the

ture termination.

CRB for the trilinear decomposition model is the inherent per-

Even though an analytical verification of these findings would mutation and scale ambiguity. To derive a meaningful CRB, we

be highly desirable from a theoretical standpoint, it seems to be assume that the first row of and is fixed (or normalized)

very difficult to carry through without making unrealistic as- to (this takes care of scale ambiguity) and further

sumptions, due to the iterative nature of ALS. The above Monte assume that the first row of is known and consists of distinct

Carlo results are perhaps sufficient from an applications view- elements (which subsequently resolves the permutation ambi-

point. guity). This way, the number of unknown complex parameters

is instead of . Similar to [3] and

VII. CONCLUSION [28], to simplify the CRB derivation, it is useful to introduce the

Low-rank decomposition of multidimensional arrays (in equivalent complex parameter vector

dimensions) has the remarkable property of uniqueness on

the basis of low-rank structure alone. It has found extensive

applications in diverse areas, including signal processing and

communications. This paper has contributed pertinent CRBs for Let us write the log-likelihood function as

low-rank decomposition of 3-D and 4-D arrays. Aside from per-

formance evaluation, these can be used as design tools for diver-

sity tradeoff studies. Easy-to-check necessary uniqueness con-

ditions for low-rank decomposition of multidimensional arrays

have also been developed. Finally, the performance of ALS al-

gorithms that are commonly used to compute low-rank decom-

position of multidimensional arrays has been assessed relative

to the respective CRBs. It was demonstrated by means of Monte

Carlo simulation that for typical problem sizes, ALS stays close

to what is theoretically achievable performance-wise. (19)

CRB FOR LOW-RANK DECOMPOSITION OF 3D ARRAYS (see, e.g., [18])

Consider an three-way array with typical element

(20)

(15)

Taking partial derivatives of with respect to the unknown

parameters, we obtain

where is i.i.d. (in , and ) zero mean, circularly sym-

metric complex Gaussian noise, of variance . Unfolding as in

(4)(6), define (21a)

(16) (21b)

(17)

(21c)

(18)

2082 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 9, SEPTEMBER 2001

(21d)

(21e)

(21f)

where is the th unit coordinate vector. where the covariance matrix of and is given by

Although the computation of the FIM may look cumbersome, (25), shown at the bottom of the page. In (25), the row index and

some simplifications are possible. Note that although is column index of nonzero elements of the covariance matrix are

independent in three dimensions, i.e., indicated explicitly. and

can be obtained similarly.

(22) Furthermore, from (20) and (24), can be written as

fact that (e.g., cf. [27])

(23)

with obvious notation. Let

(24) Then

. Given this block diagonal structure, we need to compute The CRB on the variance of any unbiased estimator of the

and invert the submatrix . Based on (22) and (23), it can be trilinear model (15) is the inverse of , which is given in terms

shown that the elements of can be computed as of

CRB CRB CRB

CRB CRB CRB (27)

CRB CRB CRB

To find the CRB of the unknown elements of , we use the

following formula for the inverse of a partitioned Hermitian ma-

trix:

CRB

..

.

..

. (25)

..

.

..

.

LIU AND SIDIROPOULOS: CRAMRRAO LOWER BOUNDS FOR LOW-RANK DECOMPOSITION 2083

CRB FOR LOW-RANK DECOMPOSITION OF 4D ARRAYS

Consider an array with typical element

(28)

for , , , ,

with . is i.i.d. (in , , , and )

zero mean, complex Gaussian noise of variance . Define ,

, , and as in Section II, and unfold in four equivalent

matrix representations:

(29)

(30)

(31)

The complex FIM is

(32)

(33)

Correspondingly, we can write the likelihood function in four

equivalent ways as

The partial derivatives of with respect to the unknown pa-

rameters are given by

(34)

Similar to (24), the FIM in (33) can be written as

(35)

whose size is .

where denotes the th column of and similarly for , The elements of can be computed as in

and . As mentioned in Section III, for the purpose of removing

sale and permutation ambiguity to obtain a meaningful bound,

we assume that the first row of , , , and are known.

Therefore, the unknown complex

parameter vector is (36a)

(36b)

2084 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 9, SEPTEMBER 2001

quadrilinear model (28) is then given by the inverse of (35):

CRB

(36c)

omitted here for brevity.

(36d) APPENDIX C

CRB FOR LOW-RANK DECOMPOSITION OF 3-D ARRAYS WITH

VANDERMONDE STRUCTURE IN ONE DIMENSION

Consider the three-way array in (15), and assume

without loss of generality that is Vandermonde:

(36e)

(37)

..

.

(36f)

complex parameter vector is given by

(36g)

The partial derivatives of with respect to and are

(36h)

same as in (21b), (21c), (21e), and (21f). The remaining proce-

(36i)

dures to obtain the CRB are similar to those in Appendix A and,

hence, are omitted for brevity.

APPENDIX D

CRB FOR REAL-PARAMETER LOW-RANK DECOMPOSITION OF

(36j) 3-D ARRAYS

The real case yields simpler CRB expressions, due to the fact

It is cumbersome to write out the covariance matrix of

that the noise covariance matrix can be written in convenient

and , but its computation can be done in the following

closed form, and an alternative scalar computation approach can

simple way. First, construct an auxiliary four-way array U and

be adopted. Consider the three-way array with

fill it with integers from 1 to as

typical element

in four different matrix forms, say, , , where is zero mean real-valued Gaussian i.i.d. in , , of

and corresponding to (29)(32), re- variance .

spectively. Now, to obtain , we can simply The likelihood function of is given by

compare and . If and contains

the same integer, then ; otherwise,

. In this way, we can easily obtain

all necessary covariance matrices.

LIU AND SIDIROPOULOS: CRAMRRAO LOWER BOUNDS FOR LOW-RANK DECOMPOSITION 2085

Thus, the log-likelihood function is Then, it can be shown that the FIM in (38) is given by

(42)

containing replicas of , and , are con-

structed similarly, i.e.,

where

given by

where denotes an identity matrix. ,

, and in (42) are given by (43)(45), as follows:

(38)

.. .. .. (43)

. . .

First, we calculate the derivatives of with respect to ,

, and

.. .. .. (44)

(39) . . .

(40) .. .. ..

. . . (45)

(41)

The CRB on the variance of any unbiased estimator is then

given by the inverse of (42), provided it exists.

From (39)(41), we can obtain

ACKNOWLEDGMENT

The authors would like to thank an anonymous reviewer for

constructive comments that helped improve the quality and

readability of this paper.

REFERENCES

[1] S. Basu and Y. Bresler, The stability of nonlinear least squares problems

and the CramrRao bound, IEEE Trans. Signal Processing, vol. 48,

pp. 34263436, Dec. 2000.

[2] J. Beltran, J. Guiteras, and R. Ferrer, Three-way multivariate calibration

procedures applied to high-performance liquid chromatography coupled

with fast-scanning spectrometry (HPLC-FSFS) detection. Determina-

tion of aromatic hydrocarbons in water samples, Anal. Chem., vol. 70,

pp. 19491955, 1998.

[3] A. van den Bos, A Cramr-Rao lower bound for complex parameters,

IEEE Trans. Signal Processing, vol. 42, p. 2859, Oct. 1994.

[4] J. W. Brewer, Kronecker products and matrix calculus in system

theory, IEEE Trans. Circuits Syst., vol. CAS-25, pp. 772781, Sept.

1978.

[5] R. Bro, PARAFAC: tutorial and applications, Chemometr. Intell. Lab.

Define the matrices Syst., vol. 38, pp. 149171, 1997.

[6] J. D. Carroll and J. Chang, Analysis of individual differences in mul-

N

tidimensional scaling via an -way generalization of Eckart-Young

decomposition, Psychometrika, vol. 35, pp. 283319, 1970.

[7] R. B. Cattell, Parallel proportional profiles and other principles for de-

termining the choice of factors by rotation, Psychometrika, vol. 9, pp.

267283, 1944.

[8] R. A. Harshman, Foundations of the PARAFAC procedure: Model and

conditions for an explanatory multi-mode factor analysis, UCLA

Working Papers in Phonetics, vol. 16, pp. 184, Dec. 1970.

[9] , Determination and proof of minimum uniqueness conditions for

PARAFAC1, UCLA Working Papers Phonetics, vol. 22, pp. 111117,

1972.

2086 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 9, SEPTEMBER 2001

[10] , PARAFAC2: mathematical and technical notes, UCLA Working [28] S. F. Yau and Y. Bresler, A compact CramrRao bound expression

Papers Phonetics, vol. 22, pp. 3344, 1972. for parametric estimation of superimposed signals, IEEE Trans. Signal

[11] R. A. Harshman and M. E. Lundy, The PARAFAC model for three-way Processing, vol. 40, pp. 12261229, May 1992.

factor analysis and multidimensional scaling, in Res. Methods Multi-

mode Data Anal., H. G. Law, C. W. Snyder, Jr, J. Hattie, and R. P. Mc-

Donald, Eds. New York: Praeger, 1984, pp. 122215.

[12] S. Kay, Fundamentals of Statistical Signal Processing. Upper Saddle

River, NJ: Prentice-Hall, 1993, vol. I, Estimation Theory.

[13] C. G. Khatri and C. R. Rao, Solutions to some functional equations

Xiangqian Liu received the B.S. degree from

and their applications to characterization of probability distributions,

Beijing Institute of Technology, Beijing, China,

Sankhya, Se. A, vol. 30, pp. 167180, 1968.

in 1997 and the M.S. degree from University of

[14] J. B. Kruskal, Three-way arrays: rank and uniqueness of trilinear de-

Virginia, Charlottesville, in 1999, both in electrical

compositions, with application to arithmetic complexity and statistics,

engineering. He is currently pursuing the Ph.D.

Lin. Algebra Applicat., vol. 18, pp. 95138, 1977.

[15] , Rank, decomposition, and uniqueness for 3-way and -way N degree with the Department of Electrical and

Computer Engineering, University of Minnesota,

arrays, in Multiway Data Analysis, R. Coppi and S. Bolasco,

Minneapolis.

Eds. Amsterdam, The Netherlands: North-Holland, 1988, pp. 718.

His research interests are in the area of signal pro-

[16] J. K. Lee, R. T. Ross, S. Thampi, and S. E. Leurgans, Resolution of

cessing for communications with focus on wireless

the properties of hydrogen-bonded tyrosine using a trilinear model of

multiuser communications.

fluorescence, J. Phys. Chem., vol. 96, pp. 91586192, 1992.

[17] X. Liu and N. Sidiropoulos, PARAFAC methods for blind beam-

forming: multilinear ALS performance and CRB, in Proc ICASSP,

Istanbul, Turkey, June 59, 2000.

[18] C. R. Rao, Minimum variance and the estimation of several parame-

ters, in Proc. Cambridge Phil. Soc., vol. 43, 1946, pp. 280283.

[19] E. Sanchez and B. R. Kowalski, Generalized rank annihilation factor Nicholas D. Sidiropoulos (M92SM99) received

analysis, Anal. Chem., vol. 58, pp. 496499, 1986. the Diploma in electrical engineering from the Aris-

[20] P. H. Schonemann, An algebraic solution for a class of subjective met- totelian University of Thessaloniki, Thessaloniki,

rics models, Psychometrika, vol. 37, p. 441, 1972. Greece, and the M.S. and Ph.D. degrees in electrical

[21] N. D. Sidiropoulos and R. Bro, On the uniqueness of multilinear de- engineering from the University of Maryland,

N

composition of -way arrays, J. Chem., vol. 14, pp. 229239, May College Park (UMCP), in 1988, 1990, and 1992,

2000. respectively.

[22] , On communication diversity for blind identifiability and the From 1988 to 1992, he was a Fulbright Fellow

N

uniqueness of low-rank decomposition of -way arrays, in Proc and a Research Assistant with the Institute for

ICASSP, Istanbul, Turkey, June 59, 2000. Systems Research (ISR), UMCP. From September

[23] , PARAFAC techniques for signal separation, in Signal Pro- 1992 to June 1994, he served his military service

cessing Advances in Wireless Communications, P. Stoica, G. Giannakis, as a Lecturer with the Hellenic Air Force Academy. From October 1993 to

Y. Hua, and L. Tong, Eds. Upper Saddle River, NJ: Prentice-Hall, June 1994, he was a Member of Technical Staff, Systems Integration Division,

2000, vol. 2, ch. 4. G-Systems Ltd., Athens, Greece. He has held Postdoctoral (from 1994 to

[24] N. D. Sidiropoulos, R. Bro, and G. B. Giannakis, Parallel factor analysis 1995) and Research Scientist (from 1996 to 1997) positions at ISR-UMCP,

in sensor array processing, IEEE Trans. Signal Processing, vol. 48, pp. before joining the Department of Electrical Engineering, University of Virginia

23772388, Aug. 2000. , Charlottesville, in July 1997 as an Assistant Professor. He is currently an

[25] N. D. Sidiropoulos, G. B. Giannakis, and R. Bro, Blind PARAFAC Associate Professor with the Department of Electrical and Computer Engi-

receivers for DS-CDMA systems, IEEE Trans. Signal Processing, vol. neering, University of Minnesota, Minneapolis. His current research interests

48, pp. 810823, Mar. 2000. are primarily in multi-way analysis and its applications in signal processing

[26] N. D. Sidiropoulos and X. Liu, Identifiability results for blind beam- for communications and networking.

forming in incoherent multipath with small delay spread, IEEE Trans. Dr. Sidiropoulos is a member of the Signal Processing for Communications

Signal Processing, vol. 49, pp. 228236, Jan. 2001. Technical Committee (SPCOM-TC) of the IEEE SP Society and currently

[27] P. Stoica and A. Nehorai, MUSIC, maximum likelihood, and serves as Associate Editor for IEEE TRANSACTIONS ON SIGNAL PROCESSING

Cramr-Rao bound, IEEE Trans. Acoust., Speech, Signal Processing, and the IEEE SIGNAL PROCESSING LETTERS. He received the NSF/CAREER

vol. 37, pp. 720741, May 1989. award for the Signal Processing Systems Program in June 1998.

- Chapter 09Uploaded byMuhammad Asad Ali
- Partial Least Squares Regression a TutorialUploaded byazhang576
- MatricesUploaded byAmanda Arnachellum
- 2nd PU Maths Model QP 2Uploaded byPrasad C M
- UT Dallas Syllabus for engr2300.002.11f taught by Naofal Al-Dhahir (nxa028000)Uploaded byUT Dallas Provost's Technology Group
- Ch 4 DeterminantUploaded byAnshuman Singh
- WAVELET_PSO-BASED_LQR_ALGORITHM_FOR_OPTI.pdfUploaded byFilipus Ivan Denata
- AT-2016-17-C-XI _PAPER-1_-AT+PCM.pdfUploaded byhiralalnhpc
- ANALISIS SEUDOTRIDIMENSIONALUploaded byronyro
- RPT Matematik T5Uploaded byorientus86
- lucerogaytan magic numberUploaded byapi-288337394
- SM1421701_SolUploaded by213698 ktr.et.ece.15
- nmf-modUploaded byjuan perez arrikitaun
- hw1Uploaded byYehh
- 12th First Term Paper Maths 2011-12-1Uploaded byvishaljalan
- mazziaUploaded byBarracuda Benary
- Space-Time Behavior-based Correlation-Or-How to Tell if Two Underlying Motion Fields Are Similar Without Computing Them - Shechtman Irani - IEEE Transactions on Pattern Analysis and Machine Intelligence - 2007Uploaded byzukun
- assessmnet 02Uploaded byLekhani Dasanayake
- Book Linear34546456Uploaded byRajaDurai Ramakrishnan
- Ans 1 SmoothUploaded byChristina Jones
- Computation of E FieldsUploaded bythgnguyen
- floquetUploaded byThalia Juarez
- Synopsis Vectors MatricesUploaded byravb12-1
- 1 - LAB No 1Uploaded byHabib Faridoon
- sample paper-1617-c-10-paper-1Uploaded byapi-306786535
- Incropera_solutUploaded byLuigiBana
- Linear Systems Ch4 [Compatibility Mode]Uploaded byUmair Zulfiqar
- ISS-12-02Uploaded byslogeshwari
- gate-2014-syllabus-for-computer-science-and-information-technology.pdfUploaded byAnshu Gaurav
- differensialUploaded byfntvrs

- Hindi 4 kids, AlphabetUploaded byAnki Ankita
- 09 Chapter 3Uploaded byUma Mageshwari
- 12 Chapter 3Uploaded byUma Mageshwari
- CATIA_labUploaded byUma Mageshwari
- ME-CAD-CAM-R-2013Uploaded byUma Mageshwari
- CATIA_Labsheets.pdfUploaded byNagaraj Kaluvacharla
- Orthographic ProjectionUploaded byssaygun
- PS-MPU-6000AUploaded byUma Mageshwari
- RM.pdfUploaded byUma Mageshwari
- abstractUploaded byUma Mageshwari
- art-3A10.1007-2FBF00199544Uploaded byUma Mageshwari
- Lecture 19Uploaded byUma Mageshwari
- Covariance Matrix EstimationUploaded byUma Mageshwari
- Channel Estimation for Ultra-Wideband CommUploaded byUma Mageshwari
- Geometry of CRBUploaded byUma Mageshwari
- Detection.pdfUploaded byUma Mageshwari
- 080240_8Uploaded byUma Mageshwari
- Electronics 06 00015 v2Uploaded byUma Mageshwari
- Aero 2n v2Uploaded byUma Mageshwari
- Staging and Control of Rockets and Missiles_ _ PritamashutoshUploaded byUma Mageshwari
- Notes 02Uploaded byUma Mageshwari
- Rocket Equation Derivation Notes (1)Uploaded bySrikar Ghooli
- Old Question PaperUploaded byUma Mageshwari
- 19880010925Uploaded byUma Mageshwari
- xmpl-421-1st-order-01.pdfUploaded byLourdhu Sagaya Jayakumar J
- IEEE Joint Angles Using Accelerometer and GyroscopeUploaded byUma Mageshwari
- AdaUploaded byUma Mageshwari

- Continuous Time Review CochraneUploaded bySophie Béreau
- The Circle of Life: the Heart's Journey Through the Seasons - Joyce RuppUploaded byTeresa Valladares
- D3689-90-Standard Test Method for Individual Piles Under Static Axial Tensile LoadUploaded byYazmin Lugo J
- EEET2369 Lecture 1_PresentationUploaded byGeoffrey Scarlett
- Democratic LeadershipUploaded bychoprachirag
- SAP BPC EPM ofc 10 User EnUploaded byAshok Kumar
- Abap Program Tips v3Uploaded byRoberto Martínez
- Blacklisted411.Net - Edition 4 - 2006.01.16Uploaded byjformica
- Structure Contour MapsUploaded byAbdul Moeed Kalson
- Agriculture to Agri-Tourism Revised2016!01!21Uploaded byClinton De Leon Serquiña
- Ethics and Social ResponsibilityUploaded byMaria Gabriela Popa
- The Birth of the Human Being..Your BirthUploaded byAmiarMira
- Oracle JDE Grower Management Data SheetUploaded byNguyen Tran Tuan
- ThermodynamicsUploaded byvinovictory8571
- Ch9 SolutionUploaded byRahul Patel
- supplier_auditUploaded byAmilaLakmal
- Description of CoursesUploaded byShwet Kumar
- Health Education 6 8Uploaded byIsabel Behrmann
- Solutions Manual for Mechanics and ThermodynamicsUploaded byPaduraru Giani
- HL_2_2486P 2011_10(1)[Specification of Communication Between Pt4V and LIMS]Uploaded byJose Perez Perez
- Cataract Case StudyUploaded byibheng_ling
- Data TransmissionUploaded byPrerak Dedhia
- clinical cases for MRCPCH PART 2 applied knowledge in practiceUploaded byaeyousef
- Colebrook EquationUploaded byAhmed Awadalla
- Suburbanisation and Counterurbanisation Revision NotesUploaded bysans1
- Ge Mid Idm 6000 1.Lm6000 PerformanceUploaded byChristopher Hampton
- MM CookbookUploaded bySuvendu Bishoyi
- The Innovator's SolutionUploaded bySantiago Mata-Mouros
- Business Influence on Govt & Public PolicyUploaded byansalmch
- Pipe NetUploaded bySARFRAZ ALI