(IJCSIS) International Journal of Computer Science and Information Security,Vol. 10, No. 9, September 2012
Additive Update Algorithm for Nonnegative MatrixFactorization
Tran Dang Hien
Vietnam National Universityhientd_68@yahoo.com Do Van TuanHanoi College of Commerce and TourismHanoi – Vietnamdvtuanest@gmail.com Pham Van AtHanoi University of Communicationsand Transportphamvanat83@vnn.vn
Abstract
—Nonnegative matrix factorization (NMF) is anemerging technique with a wide spectrum of potentialapplications in data analysis. Mathematically, NMF can beformulated as a minimization problem with nonnegativeconstraints. This problem is currently attracting much attentionfrom researchers for theoretical reasons and for potentialapplications. Currently, the most popular approach to solve NMFis the multiplicative update algorithm proposed by D.D. Lee andH.S. Seung. In this paper, we propose an additive updatealgorithm, that has faster computational speed than thealgorithm of D.D. Lee and H.S. Seung.
Keywords - nonnegative matrix factorization; Krush-Kuhn-Tucker optimal condition; the stationarity point; updating anelement of matrix; updating matrices;
I.
I
NTRODUCTION
Nonnegative matrix factorization approximation (NMF) isan approximate representation of a given nonnegative matrix
nxm
RV
by a product of two nonnegative matrices
nxr
RW
and
rxm
R H
:
H W V
Because
r
is usually chosen by a very small number, thesize of the matrices
W
and
H
are much smaller than
V
. If
V
is adata matrix of some object then
W
and
H
can be viewed as anapproximate representation of
V
. Thus NMF can be consideredan effective technique for representing and reducing data.Although this technique appeared only recently, it has wideapplication, such as document clustering [7, 11], data mining[8], object recognition [5] and detecting forgery [10, 12]To measure the approximation in (1.1) often use theFrobenius norm of difference matrix
nim jijijF
V WH V WH H W f
1122
))((2121),(
Thus NMF can be formulated as an optimization problemwith nonnegative constraints:
),(min
0,0
H W f
H W
Since the objective function is not convex, most methodsfail to find the global optimal solution of the problem and onlyget the stationary point, i.e. the matrix pair
(W, H)
satisfies theKrush-Kuhn-Tucker (KKT) optimal condition [2]:
0,0
bjia
H W
0))((,0))((
bjT iaT
V WH W H V WH
jbaiV WH W H
H V WH W
bjT bjiaT ia
,,,,0))((
,0))((
Where
W H W f H V WH
T
/ ),()(
H H W f V WH W
T
/ ),()(
To update the
H
or
W
(the remaining fixed) often use thegradient direction reverse with a certain appropriate steps, sothat a reduction in the objective function, on the other still toensure non-negative of
H
and
W
. Among the known algorithmssolve (1.3) must be mentioned algorithm LS (DD Lee and HSSeung [6]). This algorithm is a simple calculation scheme, easyto install and gives quite good results, so now it remains one of the algorithms are commonly used [10, 12]. LS algorithm isadjusted using the following formula:
ijT T ijijijijijij
WH W AW H
H f H H
)(~
ijT T ijijijijijij
H H W H AW
W f W W
)~~~(~
1http://sites.google.com/site/ijcsis/ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,Vol. 10, No. 9, September 2012
By selecting
ij
and
ij
by the formula:
ijT ijij
WH W H
)(
ijT ijij
H H W W
)~~(
then the formula (1.5) and (1.6) become:
ijT ijT ijij
WH W AW H H
)()(~
ijT ijT ijij
H H W H AW W
)~~()~(~
The adjustment formula uses multiplication so thisalgorithm is called the method of
multiplicative update
. Withthis adjustment to ensure non-negative of
W
~
and
H
~
. In [6]prove the monotonic decrease of the objective function afteradjustment:
),()
~,~(
H W f H W f
This algorithm LS has the advantages of simple easy toimplement on the computer. However, the coefficients
ij
and
ij
are selected in a special way should not reach the minimumin each adjustment. This limits the speed of convergence of thealgorithm.To improve the convergence speed, E.F. Gonzalez and Y.Zhang has improved LS algorithm by using a coefficient foreach column of
H
and a coefficient for each row of
W
. In otherwords instead of (1.5) (1.6) using the following formula:
ijT T ij jijij
WH W AW H H
)(~
ijT T ijiijij
H H W AH W W
)~~(~
The coefficients
j
and
i
are calculated through thefunction:
),,(
xb Ag
(A is the matrix. B and x is the vector). This function isdefined as follows:
)(
Axb Aq
T
q Ax A x p
T
) /(.
Where the symbol “./” and “
” denote component-wisedivision and multiplication, respectively. Then calculate
),,(
xb Ag
by the formula:
0:max99.0,min
p x Ap A pq p
T T T
The coefficients
j
and
i
are determined by the function
g(A,b,x)
as follows:
n j H V W g
j j j
..1),,,(
miW V H g
T iT iT i
..1),,,(
However, the experiments showed that improvement of EFGonzalez and Y. Zhang has not really bring obvious effect.Also as remarks in [3], the LS algorithm and GZ algorithm (EFGonzalez - Y. Zhang [3]) are not guaranteed the convergenceto a stationary point. New algorithm uses addition for updating,so it is called additive update algorithm.In this paper we propose a new algorithm by updating eachelement of every matrix
W
and
H
based on the idea of nonlinear Gauss - Seidel method [4]. Also with someassumptions, the proposed algorithm ensures reachingstationary point (Theorem 2, subsection III.B). Experimentsshow that the proposed algorithm converges faster than thealgorithms LS and GZ.The content of the paper is organized as follows. In section 2,we present an algorithm to update an element of the matrix Wor H. This algorithm will be used in section 3 to construct anew algorithm for NMF (1.3). We also consider someconvergence properties of new algorithm. Section 4 presents ascheme for installing a new algorithm on a computer. Insection 5, we present experimental results comparing thecalculation speed of algorithms. Finally some conclusions aregiven in section 6.
II.
A
LGORITHM FOR UPDATING AN ELEMENT OF MATRIX
A.
Updating an element of matrix W
In this section, we consider the algorithm for updating anelement of
W
, while retaining the remaining elements of
W
and
H
. Suppose
ij
W
is adjusted by adding
parameter:
ijij
W W
~
If
W
~
is an obtained matrix, then by some matrixoperations, we have:
mbia H WH
mbiaWH
H W
jbibabab
..1,,)(
..1,,)(
)~(
So from (1.2) it follows:
)(),(),
~(
g H W f H W f
2http://sites.google.com/site/ijcsis/ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,Vol. 10, No. 9, September 2012
Where
q pg
)(21)(
2
mb jb
H p
12
mb jbib
H V WH q
1
*)(
To minimize
),~(
H W f
, one needs to define
so that
)(
g
achieves the minimum value on the condition
0~
ijij
W W
. Because
)(
g
is a quadratic function,then
can be defined as follows:
0,0),,max(
0,0
q pqqw pqq
ij
Formula (2.6) always means because if
q
0
then by (2.4)we have
p>0
From (2.3) and (2.6), we get:
0)(
g
, if (q = 0) or (q>0 and
ij
W
=0) (2.7.a)
0)(
g
, otherwise (2.7.b)By using update formulas (2.1) and (2.6), the monotonousdecrease of the objective function
f(W,H)
is confirmed in thefollowing lemma .
LEMMA 1: If conditions KTT are not satisfied at
ij
W
, then:
),(),
~(
H W f H W f
Otherwise:
W W
~
Proof.
From (2.4), (2.5) it follows
ijT
H V WH q
))((
Therefore, if conditions KTT (1.4) are not satisfied at
ij
W
,then properties
0,0,0
qW qW
ijij
cannot occur simultaneously. From this andbecause
0
ij
W
,
it follows that case (2.7.a) cannot happen. Socase (2.7.b) must occur and we have
0)(
g
. Therefore,from (2.2) we obtain
),(),
~(
H W f H W f
Conversely, if (2.8) is satisfied, it means that:
q=0
or
q>0
and
ij
W
= 0. So from (2.6), it follows
0
. Therefore, by(2.1) we have
ijij
W W
~
Thus lemma is proved.
B.
Updating an element of matrix H
Let
H
~
be matrix obtained from the update rule:
ijij
H H
~
where
is defined by the formulas:
naai
W u
12
naajai
V WH W v
1
)(
0,0),,max(
0,0
vuvv H uvv
ij
By the same discussions used in lemma 1, we have
LEMMA 2: If conditions KTT are not satisfied at H
ij
, then:
),()
~,(
H W f H W f
Otherwise:
H H
~
III.
T
HE
P
ROPOSED
A
LGORITHM
A.
Updating matrices W and H
In this section we consider the transformation
T
from
(W, H)
to (
W
~
,
H
~
) as follows:
Modify elements of
W
by subsection II.A
3http://sites.google.com/site/ijcsis/ISSN 1947-5500
Search History:
Result 00 of 00
00 results for result for
p.
Additive Update Algorithm for Nonnegative Matrix Factorization
Nonnegative matrix factorization (NMF) is an emerging technique with a wide spectrum of potential applications in data analysis. Mathematically, NMF can be formulated as a minimization problem with nonnegative constraints. This problem is currently attracting much…
(More)
119
Reads
0
Readcasts
0
Embed Views
More From This User
Notes
Load more
