This action might not be possible to undo. Are you sure you want to continue?

Andrew would be delighted if you found this source material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. PowerPoint originals are available. If you make use of a significant portion of these slides in your own lecture, please include this message, or the following link to the source repository of Andrew’s tutorials: http://www.cs.cmu.edu/~awm/tutorials . Comments and corrections gratefully received.

**K-means and Hierarchical Clustering
**

Andrew W. Moore Professor School of Computer Science Carnegie Mellon University

www.cs.cmu.edu/~awm awm@cs.cmu.edu 412-268-7599

Nov 16th, 2001

Copyright © 2001, Andrew W. Moore

Some Data

This could easily be modeled by a Gaussian Mixture (with 5 components) But let’s look at an satisfying, friendly and infinitely popular alternative…

Copyright © 2001, 2004, Andrew W. Moore

K-means and Hierarchical Clustering: Slide 2

1

Suppose you transmit the coordinates of points drawn randomly from this dataset. You can install decoding software at the receiver. You’re only allowed to send two bits per point. It’ll have to be a “lossy transmission”. Loss = Sum Squared Error between decoded coords and original coords. What encoder/decoder will lose the least information?

Copyright © 2001, 2004, Andrew W. Moore

Lossy Compression

K-means and Hierarchical Clustering: Slide 3

Suppose you transmit the coordinates of points Break into a grid, drawn decode each bit-pair randomly from this dataset.

each You can install decoding grid-cell software at the receiver. as the middle of

Idea One

You’re only allowed to send two bits per point. It’ll have to be a “lossy transmission”. Loss = Sum Squared Error between decoded coords and original coords. What encoder/decoder will lose the least information?

Copyright © 2001, 2004, Andrew W. Moore

00

01

10

11

Any

eas? er Id Be t t

K-means and Hierarchical Clustering: Slide 4

2

Moore K-means and Hierarchical Clustering: Slide 6 3 . Andrew W. Ask user how many clusters they’d like. Andrew W. centroid of all data in Idea Two You’re only allowed to send two bits per point. Loss = Sum Squared Error between decoded coords and original coords. decode randomly from thiseach bit-pair as the dataset. Moore 00 01 10 11 as? r I de urthe ny F A K-means and Hierarchical Clustering: Slide 5 K-means 1. that grid-cell You can install decoding software at the receiver. (e. What encoder/decoder will lose the least information? Copyright © 2001.Suppose you transmit the Break into coordinates of points drawna grid.g. 2004. 2004. It’ll have to be a “lossy transmission”. k=5) Copyright © 2001.

K-means 1. 2004. Andrew W. (Thus each Center “owns” a set of datapoints) Copyright © 2001. Moore K-means and Hierarchical Clustering: Slide 7 K-means 1. 2004. Each datapoint finds out which Center it’s closest to. k=5) 2. (e.g. Ask user how many clusters they’d like. (e. Randomly guess k cluster Center locations Copyright © 2001. k=5) 2. Randomly guess k cluster Center locations 3. Andrew W. Moore K-means and Hierarchical Clustering: Slide 8 4 . Ask user how many clusters they’d like.g.

Randomly guess k cluster Center locations 3. Randomly guess k cluster Center locations 3. Each datapoint finds out which Center it’s closest to. …Repeat until terminated! Copyright © 2001. k=5) 2.g. Each datapoint finds out which Center it’s closest to. k=5) 2. (e. 4. Andrew W. Moore K-means and Hierarchical Clustering: Slide 9 K-means 1. Each Center finds the centroid of the points it owns Copyright © 2001. Each Center finds the centroid of the points it owns… 5. Andrew W. Moore K-means and Hierarchical Clustering: Slide 10 5 . 4. …and jumps there 6. 2004. Ask user how many clusters they’d like. (e.K-means 1. 2004. Ask user how many clusters they’d like.g.

Andrew W.autonlab. Accelerating Exact k-means Algorithms with Geometric Reasoning. Moore K-means and Hierarchical Clustering: Slide 11 K-means continues … Copyright © 2001. 2004.html) Copyright © 2001. Andrew W. Moore K-means and Hierarchical Clustering: Slide 12 6 .org/pap. Conference on Knowledge Discovery in Databases 1999.K-means Start Advance apologies: in Black and White this example will deteriorate Example generated by Dan Pelleg’s super-duper fast K-means system: Dan Pelleg and Andrew Moore. Proc. (KDD99) (available on www. 2004.

K-means continues … Copyright © 2001. 2004. Moore K-means and Hierarchical Clustering: Slide 14 7 . Andrew W. Moore K-means and Hierarchical Clustering: Slide 13 K-means continues … Copyright © 2001. 2004. Andrew W.

Andrew W. 2004. Moore K-means and Hierarchical Clustering: Slide 15 K-means continues … Copyright © 2001. 2004. Andrew W. Moore K-means and Hierarchical Clustering: Slide 16 8 .K-means continues … Copyright © 2001.

K-means continues … Copyright © 2001. 2004. Andrew W. Moore K-means and Hierarchical Clustering: Slide 17 K-means continues … Copyright © 2001. 2004. Moore K-means and Hierarchical Clustering: Slide 18 9 . Andrew W.

Moore K-means and Hierarchical Clustering: Slide 19 K-means terminates Copyright © 2001. 2004. 2004. Moore K-means and Hierarchical Clustering: Slide 20 10 . Andrew W.K-means continues … Copyright © 2001. Andrew W.

k] → ℜm Define… Distortion = ∑ (x i − DECODE[ENCODE(x i )]) i =1 R 2 Copyright © 2001.k] •a decoder function: DECODE : [1. Moore K-means and Hierarchical Clustering: Slide 21 Distortion Given. Andrew W. Andrew W..K-means Questions • What is it trying to optimize? • Are we sure it will terminate? • Are we sure it will find an optimal clustering? • How should we start it? • How could we automatically choose the number of centers? …. 2004.. 2004. •an encoder function: ENCODE : ℜm → [1.we’ll deal with these questions over the next few slides Copyright © 2001. Moore K-means and Hierarchical Clustering: Slide 22 11 ..

k] •a decoder function: DECODE : [1.. c2 .Distortion Given. Moore K-means and Hierarchical Clustering: Slide 24 12 . Andrew W. Andrew W. •an encoder function: ENCODE : ℜm → [1.k] → ℜm Define… Distortion = ∑ (x i − DECODE[ENCODE(x i )]) i =1 R 2 We may as well write DECODE[ j ] = c j R i =1 so Distortion = ∑ (x i − c ENCODE ( xi ) ) 2 Copyright © 2001. Moore K-means and Hierarchical Clustering: Slide 23 The Minimal Distortion Distortion = ∑ (x i − c ENCODE ( xi ) ) 2 i =1 R What properties must centers c1 . 2004. 2004. … . ck have when distortion is minimized? Copyright © 2001...

Andrew W.c 2 . Moore K-means and Hierarchical Clustering: Slide 26 13 .at the minimal distortion Copyright © 2001. c2 . … ...c k } .. ck have when distortion is minimized? (1) xi must be encoded by its nearest center ….The Minimal Distortion (1) Distortion = ∑ (x i − c ENCODE ( xi ) ) 2 i =1 R What properties must centers c1 ...at the minimal distortion Copyright © 2001.c k } . c2 .why? reduced by replacing ENCODE[xi] by the nearest center c ENCODE ( xi ) = arg min (x i − c j ) 2 c j ∈{c1 . Andrew W. 2004.. 2004... … .c 2 .why? c ENCODE ( xi ) = arg min (x i − c j ) 2 c j ∈{c1 . Moore K-means and Hierarchical Clustering: Slide 25 The Minimal Distortion (1) Distortion = ∑ (x i − c ENCODE ( xi ) ) 2 i =1 R What properties must centers c1 . ck have when distortion is minimized? (1) xi must be encoded by its nearest center Otherwise distortion could be ….

Distortion = = ∑ (x i =1 k R i − c ENCODE ( xi ) ) 2 ∑ j =1 i∈OwnedBy(c j ) ∑ (x i − c j )2 OwnedBy(cj ) = the set of records owned by Center cj .The Minimal Distortion (2) Distortion = ∑ (x i − c ENCODE ( xi ) ) 2 i =1 R What properties must centers c1 . ∂Distortion ∂c j = = = ∂ ∂c j i∈OwnedBy(c j ) i i∈OwnedBy(c j ) ∑ (x i − c j )2 −cj) −2 ∑ (x 0 (for a minimum) Copyright © 2001. 2004. 2004. Andrew W. … . Copyright © 2001. Moore K-means and Hierarchical Clustering: Slide 27 (2) The partial derivative of Distortion with respect to each center location must be zero. Andrew W. ck have when distortion is minimized? (2) The partial derivative of Distortion with respect to each center location must be zero. Moore K-means and Hierarchical Clustering: Slide 28 14 . c2 .

c2 . 2004. ck have when distortion is minimized? (1) xi must be encoded by its nearest center (2) Each Center must be at the centroid of points it owns. … . Andrew W. 2004. at a minimum: c j = Copyright © 2001. Copyright © 2001.(2) The partial derivative of Distortion with respect to each center location must be zero. Andrew W. Moore K-means and Hierarchical Clustering: Slide 30 15 . Distortion = = ∑ (x i =1 k R i − c ENCODE ( xi ) ) 2 ∑ j =1 i∈OwnedBy(c j ) ∑ (x i − c j )2 ∂Distortion ∂c j = = = ∂ ∂c j i∈OwnedBy(c j ) i i∈OwnedBy(c j ) ∑ (x i − c j )2 −cj) 1 ∑ xi | OwnedBy(c j ) | i∈OwnedBy(c j ) K-means and Hierarchical Clustering: Slide 29 −2 ∑ (x 0 (for a minimum) Thus. Moore At the minimum distortion Distortion = ∑ (x i − c ENCODE ( xi ) ) 2 i =1 R What properties must centers c1 .

c2 . ck have when distortion is not minimized? (1) Change encoding so that xi is encoded by its nearest center (2) Set each Center to the centroid of points it owns. either operation twice in succession.Improving a suboptimal configuration… Distortion = ∑ (x i − c ENCODE ( xi ) ) 2 i =1 R What properties can be changed for centers c1 . There’s no point applying either operation twice in succession. ( − of possible 2 into k groups records Distortion = nux iberc ENCODE ( x i ) ) m roids of i only a finite=1 rs are the cent So there are hich all Cente w What properties canin n. it go So if it tr applying There’s no pointied to . …And that’s K-means! of way finite number R re are only a The . … . x is encoded by itsgo to a (1) Change encoding so that i ion changes it must nearest center improved the nfigurat ch time the co So eaCenter to the centroid before. Moore K-means and Hierarchical Clustering: Slide 32 16 . ck t have ts they ow the distortion is not minimized? ration. Andrew W. But it can be profitable to alternate. it mus have when poin ges on an ite ation chan If the configur distortion. Why? Copyright © 2001. Why? Copyright © 2001. (2) Set eachiguration it’s never been to of points tually run out it owns. Moore K-means and Hierarchical Clustering: Slide 31 But it can be profitable to alternate. configurations be changed for centers c1 . …And that’s K-means! Easy to prove this procedure will terminate in a state at which neither (1) or (2) change the configuration. conf would even on forever. c2 . 2004. figurations of con Improving a suboptimal sconfiguration… R of partitioning ∑ Easy to prove this procedure will terminate in a state at which neither (1) or (2) change the configuration. Andrew W. 2004. … .

• Can you invent a configuration that has converged. 2004. Moore K-means and Hierarchical Clustering: Slide 33 Will we find the optimal configuration? • Not necessarily. Andrew W.Will we find the optimal configuration? • Not necessarily. but does not have the minimum distortion? Copyright © 2001. but does not have the minimum distortion? (Hint: try a fiendish k=3 configuration here…) Copyright © 2001. Andrew W. 2004. • Can you invent a configuration that has converged. Moore K-means and Hierarchical Clustering: Slide 34 17 .

Moore K-means and Hierarchical Clustering: Slide 36 18 . Copyright © 2001. 2004. but does not have the minimum distortion? (Hint: try a fiendish k=3 configuration here…) Copyright © 2001.Will we find the optimal configuration? • Not necessarily. 2004. each from a different random start configuration • Many other ideas floating around. Moore K-means and Hierarchical Clustering: Slide 35 Trying to find good optima • Idea 1: Be careful about where you start • Idea 2: Do many runs of k-means. • Can you invent a configuration that has converged. Andrew W. Andrew W.

2004. Andrew W. Moore Choosing the number of Centers • A difficult problem • Most common approach is to try to find the solution that minimizes the Schwarz Criterion (also related to the BIC) Distortion + λ (# parameters) log R = Distortion + λmk log R m=#dimensions Copyright © 2001. each from a different random start configuration Neat trick: Place first ideas floating around. Andrew W. Moore k=#Centers R=#Records K-means and Hierarchical Clustering: Slide 38 19 . • Many other center on top of randomly chosen datapoint.Trying to find good optima • Idea 1: Be careful about where you start • Idea 2: Do many runs of k-means. 2004. Place second center on datapoint that’s as far away as possible from first center : Place j’th center on datapoint that’s as far away as possible from the closest of Centers 1 through j-1 : K-means and Hierarchical Clustering: Slide 37 Copyright © 2001.

a good way to quantize realvalued variables into k non-uniform buckets • Used on acoustic data in speech understanding to convert waveforms into one of k categories (known as Vector Quantization) • Also used for choosing color palettes on old fashioned graphical display devices! Copyright © 2001. Moore K-means and Hierarchical Clustering: Slide 39 Single Linkage Hierarchical Clustering 1. Andrew W. 2004. Andrew W. Say “Every point is its own cluster” Copyright © 2001.Common uses of K-means • Often used as an exploratory data analysis tool • In one-dimension. Moore K-means and Hierarchical Clustering: Slide 40 20 . 2004.

Find “most similar” pair of clusters 3. Andrew W. Say “Every point is its own cluster” 2. Merge it into a parent cluster Copyright © 2001. 2004. 2004. Andrew W. Moore K-means and Hierarchical Clustering: Slide 42 21 . Moore K-means and Hierarchical Clustering: Slide 41 Single Linkage Hierarchical Clustering 1. Say “Every point is its own cluster” 2. Find “most similar” pair of clusters Copyright © 2001.Single Linkage Hierarchical Clustering 1.

Moore K-means and Hierarchical Clustering: Slide 44 22 . Repeat Copyright © 2001. Merge it into a parent cluster 4.Single Linkage Hierarchical Clustering 1. Find “most similar” pair of clusters 3. Andrew W. Say “Every point is its own cluster” 2. Moore K-means and Hierarchical Clustering: Slide 43 Single Linkage Hierarchical Clustering 1. Merge it into a parent cluster 4. Andrew W. 2004. Say “Every point is its own cluster” 2. Find “most similar” pair of clusters 3. Repeat Copyright © 2001. 2004.

Moore K-means and Hierarchical Clustering: Slide 46 23 . or hierarchy of datapoints (not shown here) Copyright © 2001. or taxonomy. Copyright © 2001. Merge it into a parent cluster 4. Andrew W. Say “Every point is its own cluster” 2. Andrew W. just cut the (k-1) longest links • There’s no real statistical or informationtheoretic foundation to this. 2004.Single Linkage Hierarchical How do we define similarity Clustering between clusters? • Minimum distance between points in clusters (in which case we’re simply doing Euclidian Minimum Spanning Trees) • Maximum distance between points in clusters • Average distance between points in clusters You’re left with a nice dendrogram. 2004. Moore 1. Makes your lecturer feel a bit queasy. Repeat…until you’ve merged the whole dataset into one cluster K-means and Hierarchical Clustering: Slide 45 Also known in the trade as Hierarchical Agglomerative Clustering (note the acronym) Single Linkage Comments • It’s nice that you get a hierarchy instead of an amorphous collection of groups • If you want k groups. Find “most similar” pair of clusters 3.

Moore K-means and Hierarchical Clustering: Slide 47 24 . 2004.What you should know • All the details of K-means • The theory behind K-means as an optimization algorithm • How K-means can get stuck • The outline of Hierarchical clustering • Be able to contrast between which problems would be relatively well/poorly suited to Kmeans vs Gaussian Mixtures vs Hierarchical clustering Copyright © 2001. Andrew W.

Sign up to vote on this title

UsefulNot useful- 2001 - K-Means and Hierarchical Clustering (Slides)
- Cluster.ppt
- l00-2x2
- 545 Project
- 02 Data Mining-Partitioning method
- 2002 Spring CS525 Lecture 2
- Ki2 s07 Clustering Algorithms
- Clustering Lecture
- Test
- k-means
- Ontology Based Document Clustering Using Mapreduce
- DCBOR
- 10.1.1.125
- Performance Comparison of Various Clustering Algorithm
- 10.1.1.7.8739
- Speeded-Up Bag-of-Words Algorithm for Robot Localisation through Scene Recognition
- Review on Density-based Clustering - DBSCAN, DenClue & GRID
- lesson6-Clustering1_Hirarchical
- Clustering Algorithms
- 621-week2
- Hierarchical Clustering With Multi view point Based Similarity Measure
- VOL2I1P5
- Silhouette
- 346-1343-1-PB
- Cluster Analysis
- Literature
- Lecture# 4, 5 & 6-2016
- Lecture_04_Image_Search
- Clustering of Structured Spatial Object
- An Efficient And Empirical Model Of Distributed Clustering
- kmeans11