Professional Documents
Culture Documents
factorization
Anjusha C
MA18M008
Department of Mathematics
IIT Madras
15/05/2020
What is topic modelling?
Souce:
http://derekgreene.com/slides/topic-modelling-with-scikitlearn.pdf
NMF
I K-L divergence :
P Vij
D(V ||WH) = ij (Vij log (WH) ij
− Vij + (WH)ij )
Method
I Gradient descent
I Naive GD : No guarantee of convergence
I Problem convex on only W or H at a time and not on both
simultaneously.
I So, have to optimize W keeping H fixed and then optimize H
keeping W fixed and keep alternating until convergence.
(Alternating Least Squares approach)
Lee Sung multiplicative updates
I Put forward by Daniel D Lee and H Sebastian Sung
(Link : Lee-Sung paper)
I These G-D updates are guaranteed to converge
I Frobenius norm update :
(W T V )aµ
Haµ = Haµ
(W T WH)aµ
(VH T )ia
Wia = Wia
(WHH T )ia
I K-L divergence update :
P
Wia Viµ /(WH)iµ
i
Haµ = Haµ P (1)
P k Wka
µ Haµ Viµ /(WH)iµ
Wia = Wia P (2)
ν Haν
Implementation
Dataset : ’20 newsgroups’ dataset is used. Newsgroups are
discussion groups on Usenet, which was popular in the 80s and 90s
before the web really took off. This dataset includes 18,000
newsgroups posts with 20 topics.
The ’show topics’ function picks up the top 8 words (words with
highest frequency in each topic vector). The words with the highest
frequencies are the ones that are specific to the topic.
The rows are the topics
The time taken is approximately 5.92 s.
Lee-Sung update : Frobenius norm
The
time taken is 21.27 s.
Error plot for Frobenius norm
Lee-Sung update : K-L divergence
The time taken is 54.04 s.
Error plot for Frobenius norm
References