This action might not be possible to undo. Are you sure you want to continue?

**Lecture 20 Clustering to form semantic concepts – Linguistic Labels
**

SCE, NTU, Singapore

Overview

• • • • • • • Interpretability of fuzzy representation. What is clustering for semantic labels? Histogram analysis LVQ (Linear Vector Quantization) FCM (Fuzzy C-Means) ( y ) FKP (Fuzzy Kohonen Partitioning) PFKP (Pseudo Fuzzy Kohonen Partitioning)

**Semantic Label Clustering
**

• Semantic properties of a linguistic variable

– linguistic variable L: (L T(L) U G M) (L, T(L), U, G,

where L is the name of the variable; T(L) is the linguistic term set of L; U is a universe of discourse; G is a syntactic rule which generates T(L); and M is a semantic rule that associates each T(L) with its meaning.

**– Each linguistic term set is characterized by a fuzzy
**

set which is described using a membership function

1 0.8

1 0.8

μT(x)

μG(x)

0.6 0.4 0.2 0 0 2 4 x 6 8 10

0.6 0.4 0.2 0 0 2 4 x 6 8 10

Trapezoidal MF

Gaussian MF

medium large very large universe of discourse 1 0.8 U=[0. 100] of the base variable x μT(x) very small small 0.Example of a Linguistic Variable • • • • • linguistic variable x named L=“performance” five linguistic terms g where T(L)={“very small”.2 02 0 0 ≺ 20 40 60 x (performance) 80 100 .4 0. “large” and “very large”}.6 0. Semantic assignment M is shown in the figure – normal and convex Semantic ordering ordering such that “very small”≺ “small”≺ “medium” ≺ “large” ≺ “very large”. “medium”. “small”.

μ X i ( z ) ( ) X1 ≺ X 2 Xj ≺ Xn denotes X1 precedes X2 .Criteria of Interpretability p y • Coverage: MFs cover the entire universe of discourse • Normalised: if ∃ x∈Xi such that μX ( x) = 1 i • Convex: • Ordered: X1 ≺ X 2 x ≤ y ≤ z ⇒ μ X i ( y ) ≥ min μ X i ( x ) .

DIC (Tung and Quek) etc.Clustering • Clustering is a method that organizes patterns into clusters such that patterns within a cluster are more similar to each other than patterns in other clusters. MLVQ (Ang and Quek). etc . • When the crisp partition in classical clustering analysis is replaced with a fuzzy partition or a fuzzy pseudo-partition. it is referred to as fuzzy clustering • E Examples: LVQ (K h l (Kohonen). FCM (B d k) ) (Bedzek).

Example: Particle Classification • Particles on an air filter P1: P2: P3: 10/7/2008 7 .

Histogram Analysis P1: P1 P2: P2 P3: P3 P1 Num mber P2 Num mber P2 P3 Area 10/7/2008 Area 8 .

Histogram Analysis P1: P1 P2: P2 P3: P3 Num mber P3 Perim meter P2 P1 P3 P2 Perimeter Area .

where m is the number of sample data entries and n is the number of features. .Sample Data Sets • Sample data is divided into two disjoint sets: – Design set (or training set) is used for designing a classifier – Test set (or cross validation set) is used for cross-validation evaluating the obtained classifier • Sample data is usually represented by an m by (n+1) matrix.

Sample data set Features Area 3 5 4 7 12 15 14 17 14 13 15 12 … Perimeter 6 7 4 6 11 10 12 13 19 20 22 18 … Class Class P1 P1 P1 P1 P2 P2 P2 P2 P3 P3 P3 P3 … Design set: Odd-indexed entries Test set: Even-indexed entries .

Flowchart for Histogram Analysis General flowchart: Feature extraction Data D t reduction Probability estimate Particle example: From image to features f t None Histogram analysis .

Histogram Analysis • Number of intervals vs number of data vs. points 50 samples p from a Gaussian distribution 3 bins 10 bins 25 bins .

the total area must be unity y – Can be used in any number of features.Histogram Analysis g y • Properties: – One of those nonparametric techniques which do not require explicit use of density functions – Dilemma between no. but subjected to curse of dimensionality . of intervals is equal to the square root of no. of points – Rule of thumb: no. of intervals vs. no. of points – Intervals may be unequally spaced – To convert to density functions.

Kernel and Window Estimators σ = 0.1 σ = 0.3 .

Bowman and A. Azzalini. New York:Oxford University Press. 1997. Applied Smoothing Techniques for Data Analysis: The Kernel Approach with S-Plus Illustrations.Kernel and Window Estimators • Properties: – Also known as Parzen estimator – Its computation is similar to convolution – Can be used in multi-features estimation – Width is found by trial and error normal optimal smoothing strategy σ denotes the standard deviation of the distribution h opt ⎛ 4 ⎞ =⎜ ⎟ σ ⎝ 3n ⎠ 1 5 W. .

w Repeated until the weights are w xi v w forced to stabilize through the . v x1 The winner. . whose weight has . . . g . the minimum distance from the . w w those of its neighbors . w winner . x2 w v input. updates its weights and w .Learning Vector Quantization • • • LVQ are unsupervised neural networks that determine the weights for cluster centers in an iterative and sequential manner Each output neuron has a weight vector vj that is adjusted during learning. specification of a learning rate . 1 j1 j j2 ji jn m2 mi m1 m mn y1 yj • ym xn input layer output layer vc yc .

c ⎧v(jT ) +α (T ) (x − v(jT ) ) if j = i ⎪ v(jT +1) = ⎨ v(jT ) if j ≠ i ⎪ ⎩ ||x y|| ||x-y|| is the euclidean distance c is the number of distance. clusters. x is the input vector..LVQ – Cont’d x − vi(T ) = min j ( x − v (jT ) ) for j = 1. else repeat with new vector t . vi is the ith cluster centre and α is the learning constant Pseudo Code: (1) Define number of clusters c and small terminating condition ε (2) Initialise weights (3) Determining winning neuron based on distance (4) g g ( ) Update winner: v i(T) = v i(T-1) + α i(T ) ( xk − vi(T −1) ) for i ≤ N (T) (5) Determine terminating condition.

n i =1 0 < ∑ μi ( xk ) < n for all i = 1..Fuzzy C-Means (FCM – Bezdek) y ( ) • A fuzzy pseudo-partition of a finite data set c X is defined: ∑ μ i ( xk ) = 1 for all k = 1..c 1 c k =1 n • An objective function for fuzzy clustering is (m defines the degree of fuzziness): J m (Ρ) = ∑∑ ( μi ( xk )) k =1 i =1 n c m xk − vi 2 .

c . degree of fuzziness (m) and terminating condition (ε) – Init t and pseudo parition p0 – Compute cluster centres: v1. …vi … vc vi(T ) = ∑ ( μ ( x )) k =1 n i k k =1 i k n m xk m ∑ (μ ( x )) for i = 1.. v2.FCM – Cont’d Cont d • Pseudo Code: – Define number of clusters (c).

.n – Compare distance between the partitions E= pt+1 – pt E = Ρ (T +1) − Ρ (T ) = ∑ ∑ μi(T +1) ( xk ) − μi(T ) ( xk ) i =1 k =1 c n – Terminate if E < ε is .FCM – Cont’d 2 Cont d • Pseudo Code: – Update new Pseudo Partition: ⎛ ⎜ c ⎛ xk − vi(T ) μi(T +1) ( xk ) = ⎜ ∑ ⎜ ⎜ j =1 ⎜ xk − v(jT ) ⎜ ⎝ ⎝ 2 ⎞ ⎟ 2 ⎟ ⎠ −1 1 m−1 ⎞ ⎟ ⎟ ⎟ ⎟ ⎠ for i = 1.c. k = 1..

5 1.0 5.0 1.0001 .0 6.0 1.0 3.0001 Error trend:FCM m=1.0 0.005.0 Sepal width (cm) 3.0 0.0 4.0 4.0 2.0 Petal length (cm) 180 170 160 150 50 45 40 150 140 50 45 40 35 30 35 30 25 20 15 10 5 0 0 10 20 30 40 50 60 Numb of mistakes ber 130 140 130 120 110 100 90 120 110 100 25 20 15 10 5 Num ber of mistakes e Mistakes T Total error e Mistakes 90 0 5 10 15 20 25 Total error 0 Number of iterations Number of iterations Error trend:LVQ α=0. ε=0.0 2.8.0 1.Examples: Iris data 5. δ=0.0 7.0 7.5.0 1.5 0.0 Iris virginica Iris versicolor Iris I i sentosa t 3.0 Sepal length (cm) 0.5 Petal width (cm) 2.0 4.0 8.0 6.0 2.0 0. ε=0.0 3.0 8.0 2.0 5.

batch-learning optimization algorithm. • performance depends on a good choice of weighting exponential m.Properties of FCM • FCM is a non-sequential batch learning non sequential. • Unable to perform on-line training. • Computationally and memory intensive intensive. .

5.2 0.4 2.3 6.0 Mem bership degree μ(x) m 1.3 1.0001 trapezoidal –like membership functions .0 2 2.8 0.8 0.2 3.4 1.5 5.0 4.8 0.8 3.2 0.7 5.7 Sepal length (cm) 7.2 0.5 7.8 0.3 4.4 0.4 0.6 Sepal width (cm) 4 4.9 1.1 5.Results of FCM 1.5 sentosa versicolor virginica IRIS data set .1 0. ε=0.0 1 2 3 4 5 Petal length (cm) 6 0.1 7.5 0.0 Membership degree (x) ) 1.7 Petal width (cm) 2.0 Membership degree (x) ) 0.6 0.4 0.0 Mem mbership degree (x) 0.2 0.6 0.6 0.9 6.1 2.6 0.0 0.4 0.9 0.FCM with m=1.

1 0.3 6.4 1.6 0.4 0.0001 m=2.0 Me embership degree (x) 1.1 5.5 7.9 6.2 0.0 0.2 0.6 0.0 2 2.0 1 2 3 4 5 Petal length (cm) 6 0.0 4.5 5.2 3.FCM with m=2 0 ε=0.8 3.2 02 0.6 Sepal width (cm) 4 4.6 0.7 5.8 0.1 2.0 Memb bership degree (x) Mem bers hip degree μ (x) b 1.7 Petal width (cm) 2.8 0.9 0.4 0.3 4.0 0.5 0.3 1.8 0.4 0. =0 0001 Gaussian-like membership functions .8 0.5 sentosa versicolor virginica IRIS dataset .0 Me embership degree (x) 0.1 7.0.7 Sepal length (cm) 7.9 1.4 2.Results FCM – Cont’d Cont d 1.6 0.2 02 0.4 0.

1 7.2 0.0 M em bership degr ree (x) 0.5 7.6 0.5 05 0.4 0.7 17 Petal width (cm) 2.8 0.9 0.8 0.0 2 2.4 0.2 3.02.8 0.1 01 0.3 6.7 5.Results MLVQ (Gaussian) • Width of gaussian MFi 1.6 0.0 1 2 3 4 5 Petal length (cm) 6 0.0 M em bership degr ree (x) 1.6 0.0001 .8 3.2 0.3 4.5.9 09 1.0 0.1 21 2. σ=1.0 4.7 Sepal length (cm) 7.4 0.4 1.4 0.6 Sepal width (cm) 4 4.8 0.5 25 sentosa versicolor virginica MLVQ with λ=0.0 M em bership degre (x) ee Mem bers hip degre μ (x) ee wi = vi − vclosest σ 1.1 5.3 13 1.6 0.5 5. ε=0.9 6.2 0.4 2.2 0.0 0.

4 0.2 32 3.0 M em bership deg gree (x) 0.8 0. σ=3.8 28 3.0 M em bership deg gree (x) 0.1 21 2.7 47 5.6 06 0.2 0.Results MLVQ (Gaussian) –cont’d cont d • Width of gaussian MFi 1.0 M em bership deg gree (x) 1.8 0.4 0.0 Mem bers hip deg gree μ (x) wi = vi − vclosest σ 1.4 44 1.3 13 1.5 25 sentosa versicolor virginica MLVQ with λ=0.3 6.0001 .0.9 79 0.9 6.6 06 0.9 09 1.4 0.0 1 2 3 4 5 Petal length (cm) 6 0.6 06 0.1 71 7. ε=0.2 0.02.0 0.4 0.7 55 59 63 67 Sepal length (cm) 7.7 17 Petal width (cm) 2.2 0.3 43 4.0 2 2.5 5.6 36 Sepal width (cm) 4 4.1 01 0.8 0.2 0.6 06 0.5 05 0.8 0.5 75 7.1 51 5.0 4.4 24 2.

a centroid v μ(x) 1 • This fuzzy interval is also known as a trapezoidal fuzzy number 0 α β v γ δ x . γ.Trapezoidal MF • A trapezoidal membership function μ(x) can be described by a fuzzy interval formed by four parameters (α β γ δ) and (α. β.

• MLVQ algorithm can be if x < α or x > δ ⎧0 ⎪ α−x used to derive the centroid v ⎪ if α ≤ x ≤ β • it cannot derive the ⎪ α −β μ (x) = ⎨ parameters (α. δ]=support of the fuzzy interval. • [β γ]=kernel of the fuzzy interval. ] k l f th f i t l d • [α. β. γ] where μ(x)=1 is called the kernel of μ( ) the fuzzy interval. δ) if β ≤ x ≤ γ ⎪1 ⎪ δ −x of the trapezoidal-shaped if γ ≤ x ≤ δ ⎪ ⎩ δ −γ membership function . and the subinterval [α. δ] is called the support.Trapezoidal MF – Cont’d Cont d • The subinterval [β. and [β. γ.

– η as the learning width and a small positive number ε as a stopping criterion.n • ( ) • Determine the ith cluster that the data xk belongs and Update the weights vi of the ith cluster . n=total number of data vectors i+ 1 Initialise I iti li weights: (0) i ht 2 max ( x ) − min ( x ) vi = min ( xk ) + k k k k k c for i = 1.supervised • Define: – c as the number of classes.The Fuzzy Kohonen Partition algorithm . – λ≤1/Ω as the learning constant.. k = 1.c. where Ω=number of data vectors in a cluster cluster..

The Fuzzy Kohonen Partition algorithm – supervised (cont’d) • C Compute error t cluster and diff t to l t d difference i error b t in between n iteration: e(T +1) = ∑ xk − vi(T +1) k =1 de (T +1) =e (T +1) −e (T ) • Repeat: while ¬ (de(T+1)≤ε ) – End of determining centroids .

α i = β i = δ i = γ i = ϕ i = vi(T +1) for i = 1.c i=1 i=2 i=3 • Determine the ith cluster that the data xk belongs and g Update the pseudo weights ϕi of the ith cluster ϕ i = ϕ i + η (xk − ϕ i ) ..The Fuzzy Kohonen Partition algorithm – supervised (cont’d) • I iti li Initialize – where ϕi is the pseudo weight of vi.

x k ) β i = m in ( β i .The Fuzzy Kohonen Partition algorithm – supervised (cont’d) • U d t the four points of the Trapezoidal Fuzzy Number Update th f i t f th T id l F N b (TrFN) α i = m in (α i . ϕ i ) δ i = m ax (δ i . x k ) . ϕ i ) γ i = m ax ( γ i .

5 5. ε=0.0 Membership deg gree (x) 0.2 3.6 Sepal width (cm) 4 4. η=0.2 0.5 sentosa versicolor virginica FKP with λ=0.4 0.0 4.1 2.0 Mem bership deg gree μ(x) 1.2 0.The Fuzzy Kohonen Partition algorithm – Results 1.02.1 0.2 0.3 1.1 7.8 3.0 2 2.7 Sepal length (cm) 7.9 1.4 0.8 0.5 7.3 6.7 5.4 2.6 0.4 1.0 Membership degree (x) d 0.0 0.3 4.0 Membership degree (x) d 1.4 0.9 6.6 06 0.7 Petal idth (cm) P t l width ( ) 2.0 1 2 3 4 5 Petal l P t l length ( ) th (cm) 6 0.9 0.1 5.8 0.8 0.8 0.0005 .6 0.4 0.2 0.6 06 0.5 0.

8 0.2 0.4 0.6 Sepal width (cm) 4 4.5.0 2 2.1 7.3 1.4 2.3 4.1 0.4 0.7 5.2 0.1 2.0005 .1 5.9 6.0 1 2 3 4 5 Petal length (cm) 6 0.0 M em bership degre (x) ee 0.2 0.0 4.0 Membership degre (x) ee 0.5 5.9 1.4 0.0 Membership degre (x) ee 1.7 Sepal length (cm) 7.5 0.8 0.6 0.6 0.8 0.3 6. ε=0.02.0 M em bership degre (x) ee 1.8 0.9 0.The Fuzzy Kohonen Partition algorithm – Results 1.5 sentosa versicolor virginica FKP with λ=0.5 7.2 0.4 0.2 3. η=0.8 3.4 1.6 0.7 Petal width (cm) 2.6 0.0 0.

ϕi ) ( ⎧max(δi ..c • Initialize – where ϕi is the pseudo weight of vi.Pseudo Fuzzy Kohonen Partition algorithm • PFKP operates similarly to FKP except in the determination of the trapezoidal MF parameters: ϕ i = α i = β i = δ i = γ i = vi(T +1) for i = 1 c 1.c – Determine winner centroid: j – Update pseudo weight of winner: ϕ i = ϕ i + η (x k − ϕ i ) ( ) – Update Trapezoidal Fuzzy Number p p y ⎧min(αi . xk ) for i =1 αi = ⎨ for i >1 ⎩ γi−1 βi = min(βi . • Determine TF parameters: xk − ϕi = min xk − ϕ j for j = 1..ϕi ) γi = max(γi . xk ) for i = c δi = ⎨ for i < c ⎩ βi+1 .

0 M em ber rship degree (x ) 1.9 0.0 0.6 0.Results 1. η=0.8 0.9 1.8 0.7 5.1 0.3 6.4 0.4 1.8 3. ε=0.0 M em bership degree (x) 1.3 1.2 0.4 0.8 0.6 0.5 0.5 5.2 0.9 6.4 0.1 7.4 0.2 3.0 M em bership degree (x) 0.7 Petal width (cm) 2.0005 .1 2.02.PFKP .3 4.5 7.6 0.0 M em ber rship degree (x ) 0.7 Sepal length (cm) 7.8 0.0 2 2.2 0.2 0.5 sentosa versicolor virginica PFKP with α=0.1 5.0 4.6 Sepal width (cm) 4 4.6 0.4 2.0 1 2 3 4 5 Petal length (cm) 6 0.

5 sentosa versicolor virginica with λ=0.6 0.0 Memb bership degree (x) 0.6 Sepal width (cm) 4 4.3 6.0 1 2 3 4 5 Petal length (cm) 6 0.5 5.6 0.8 0.1 7.6 0.7 Sepal length (cm) 7.0 Memb bership degree (x) 1.0 0.9 6.0 4.1 0.3 4.8 0.5 0.8 3.1 2.8 0.8 0.2 3.2 0.3 1.2 02 0.4 1.4 0.1 5.9 0. η=0.7 5.4 0. ε=0.0 Me embership degree (x) ) 1.PFKP – Results (cont d) (cont’d) 1.9 1.6 0.4 0.2 02 0.02.0 2 2.5 7.0 Me embership degree (x) ) 0.7 Petal width (cm) 2.4 2.2 0.01.0005 .4 0.

Concluding Remarks • Fuzzy Kohonen Partition (FKP) and the Pseudo Fuzzy Kohonen Partition (PFKP). • Both algorithms directly derive trapezoidal membership functions that are convex and normal from training data while the latter derive membership functions that forms a pseudo pseudopartition of the input space . were proposed to directly derive appropriate membership functions from training data.

Sign up to vote on this title

UsefulNot useful- Intelligent Computing Methods for Manufacturing Systems
- Novel Apporach for Speech Recognition by Using Self - Organized Maps
- Handwritten Script Recognition
- Rugescu AECE References
- Harvard Linguistics 116a Syllabus
- 10.1.1.93
- Padrão de venação foliar
- Glossary Semantics 2
- Semantics Final
- Erten&Tekin (2008)
- Semantics
- Graficos no Stata
- Automatic Dense Visual Semantic Mapping From Street-level Imagery
- An evloutionary algorithm approach to poetry generation.pdf
- An Evolutionay Algorith Aproach to Poetry Generation
- Stats Chap02 bluman
- Towards Best Practice for Multiword Expressions in Computational Lexicons_Boh
- Zbikowski Kuehl Review 2009
- What is Semantics
- AN ONTOLOGY BASED DEEP – MINING METHOD TO CLUSTER THE CONTENT BASED ON WEBLOG
- Communicating Your Study
- MODAL VERBS-table to Complete
- Correia, Fabrice - (Finean) Essence and (Priorean) Modality
- rfc1214
- Brandom
- Massadenella Brasil
- Flora of China - Key to Saccolabiopsis
- Semantics Based Automated Service Discovery
- 7QC Tools
- Abralin - Revista Abralin - Volume12-n1
- Clustering for Semantic labels - Wk9

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

We've moved you to where you read on your other device.

Get the full title to continue

Get the full title to continue reading from where you left off, or restart the preview.