Professional Documents
Culture Documents
Algorithm Perceptron;
Start with a randomly chosen weight vector w0;
Let k = 1;
while there exist input vectors that are
misclassified by wk-1, do
Let ij be a misclassified input vector;
Let xk = class(ij)ij, implying that wk-1xk < 0;
Update the weight vector to wk = wk-1 + xk;
Increment k;
end-while;
September 23, 2010 Neural Networks 1
Lecture 6: Perceptron Learning
Another Refresher: Linear Algebra
How can we visualize a straight line defined by an
equation such as w0 + w1i1 + w2i2 = 0?
One possibility is to determine the points where the
line crosses the coordinate axes:
i1 = 0 w0 + w2i2 = 0 w2i2 = -w0 i2 = -w0/w2
i2 = 0 w0 + w1i1 = 0 w1i1 = -w0 i1 = -w0/w1
Thus, the line crosses at (0, -w0/w2)T and (-w0/w1, 0)T.
If w1 or w2 is 0, it just means that the line is horizontal
or vertical, respectively.
If w0 is 0, the line hits the origin, and its slope i 2/ii is:
w1i1 + w2i2 = 0 w2i2 = -w1i1 i2/i1 = -w1/w2
September 23, 2010 Neural Networks 2
Lecture 6: Perceptron Learning
Perceptron Learning Example
We would like our perceptron to correctly classify the
five 2-dimensional data points below.
Let the random initial weight vector w0 = (2, 1, -2)T.
Then the dividing line crosses at
i2 (0, 1)T and (-2, 0)T.
3
-1 2
1 Let us pick the misclassified
-3 -2 -1 1 2 3
point (-2, -1)T for learning:
-1 i1 i = (1, -2, -1)T (include offset 1)
-2 1
-3
x1 = (-1)(1, -2, -1)T (i is in class -1)
class -1 x1 = (-1, 2, 1)T
class 1
September 23, 2010 Neural Networks 3
Lecture 6: Perceptron Learning
Perceptron Learning Example
w1 = w0 + x1 (let us set = 1 for simplicity)
w1 = (2, 1, -2)T + (-1, 2, 1)T = (1, 3, -1)T
The new dividing line crosses at (0, 1)T and (-1/3, 0)T.
class -1
class 1
September 23, 2010 Neural Networks 4
Lecture 6: Perceptron Learning
Perceptron Learning Example
w2 = w1 + x2
w2 = (1, 3, -1)T + (1, 0, 2)T = (2, 3, 1)T
Now the line crosses at (0, -2)T and (-2/3, 0)T.
i2 3
1 With this weight vector, the
2
1
perceptron achieves perfect
-3 -2 -1 1 2 3
classification!
-1 i1 The learning process terminates.
-2
-1 -3 In most cases, many more
iterations are necessary than in
class -1 this example.
class 1
September 23, 2010 Neural Networks 5
Lecture 6: Perceptron Learning
Perceptron Learning Results
We proved that the perceptron learning algorithm is
guaranteed to find a solution to a classification problem
if it is linearly separable.
But are those solutions optimal?
One of the reasons why we are interested in neural
networks is that they are able to generalize, i.e., give
plausible output for new (untrained) inputs.
How well does a perceptron deal with new inputs?
x0 x1 = x0 - f’(x0) x