K-Nearest Neighbors

2. K-Nearest Neighbor – Intuition (KNN)

Problem
Given two classes A, B and a new data point x. Which’ class the point x belong to?

Manh-Hung Ha ISVNU
 2. K-Nearest Neighbor – Intuition (KNN)
Problem
Given two classes A, B and a new data point x. Which’ class the point x belong to?

KNN? 1- Nearest Neighbor

• Calculate distance from point x to all the points in A and B
• Find smallest distance from point x to a point in class A and B
d3 • Check the nearest point belong to class A or B
d1 • The point x is belong to same class with the nearest point
d2
d1 < d2 < d3 …< dm
d1 = min{d1, d2, … dm}

Point “1” belong to class B => Point x belong to class B

Manh-Hung Ha ISVNU
 2. K-Nearest Neighbor – Intuition (KNN)
Problem
Given two classes A, B and a new data point x. Which’ class the point x belong to?

1-NN Distance??

d3
d1
d2

Manh-Hung Ha ISVNU
 2. K-Nearest Neighbor – Intuition (KNN)
Problem
Given two classes A, B and a new data point x. Which’ class the point x belong to?

5-NN

• Totally 5 neighbors
• 2 neighbors in class B
• 3 neighbors in class A
• New data point x belong to class A

How about KNN?

Manh-Hung Ha ISVNU
 3. KNN algorithm
Algorithm
Given multiples classes and a new data point x. Which’s class the point x belong to?
• Select the number K of the neighbors
• Calculate the distance of K number of neighbors
• Take the K nearest neighbors as per the calculated distance.
• Among these K neighbors, count the number of the data points in each category.
• Assign the new data points to that category for which the number of the neighbor is maximum.

Trial test: K = 1, 2, … N
How to find K?
Find K with the best accuracy

Manh-Hung Ha ISVNU
 3. KNN algorithm
Algorithm

• Select K (1, 2, …N)

• Calculate distance from x to all data points
• Sort K to take K nearest neighbors
• Count number of points belong to classes
• Point x is belong to the class having max
neighbor points
n
di =

Sort: d1 < d2 < d3 …< dm

Take: {d1, d2, … dk}
Count

Manh-Hung Ha ISVNU
 4. Conclusions

• KNN is a simple algorithm

• Complexity ~ numbers of data & number of features
• All features are equal importance
• Easy affected by noise data
• No parameters
• No training time, all calculation in the test time (prediction step)
• Could be used in classification and regression

Manh-Hung Ha ISVNU