12/5/2023

Non-linear Classification

Neural Networks:
Representation x2

Non-linear x1

hypotheses size
# bedrooms
# floors
Machine Learning age

Andrew Ng

What is this? Computer Vision: Car detection

You see this:

But the camera sees this:

Cars Not a car

Testing:

What is this?
Andrew Ng Andrew Ng

pixel 1 pixel 1

Learning Learning
Algorithm Algorithm
pixel 2 pixel 2

Raw image Raw image

pixel 2 pixel 2

Cars pixel 1 Cars pixel 1

“Non”-Cars Andrew Ng
“Non”-Cars Andrew Ng

1
 12/5/2023

pixel 1

Learning
Algorithm
pixel 2

Raw image 50 x 50 pixel images→ 2500 pixels

pixel 2 (7500 if RGB)

pixel 1 intensity
pixel 2 intensity

pixel 2500 intensity

Cars pixel 1 Quadratic features ( ): ≈3 million

“Non”-Cars features
Andrew Ng

Neural Networks
Neural Networks: Origins: Algorithms that try to mimic the brain.
Was very widely used in 80s and early 90s; popularity
Representation diminished in late 90s.
Recent resurgence: State-of-the-art technique for many
Neurons and applications

the brain
Machine Learning

Andrew Ng

The “one learning algorithm” hypothesis The “one learning algorithm” hypothesis

Somatosensory Cortex
Auditory Cortex

Auditory cortex learns to see Somatosensory cortex learns to see

[Roe et al., 1992] Andrew Ng [Metin & Frost, 1989] Andrew Ng

2
 12/5/2023

Sensor representations in the brain

Seeing with your tongue Human echolocation (sonar)

Haptic belt: Direction sense Implanting a 3rd eye

[BrainPort; Welsh & Blasch, 1997; Nagel et al., 2005; Constantine-Paton & Law, 2009] Andrew Ng

Neuron in the brain

Neural Networks:
Representation
Model
representation I
Machine Learning

Andrew Ng

Neurons in the brain Neuron model: Logistic unit

Sigmoid (logistic) activation function.

[Credit: US National Institutes of Health, National Institute on Aging] Andrew Ng Andrew Ng

3
 12/5/2023

Neural Network Neural Network

“activation” of unit in layer
matrix of weights controlling
function mapping from layer to
layer

If network has units in layer , units in layer , then

Layer 1 Layer 2 Layer 3 will be of dimension .
Andrew Ng Andrew Ng

Neural Networks:
Representation
Model
representation II
Machine Learning

Forward propagation: Vectorized implementation Neural Network learning its own features

Add .

Layer 1 Layer 2 Layer 3

Andrew Ng Andrew Ng

4
 12/5/2023

Other network architectures

Layer 1 Layer 2 Layer 3 Layer 4

Andrew Ng

Non-linear classification example: XOR/XNOR

, are binary (0 or 1).
Neural Networks:
Representation x2
x2

Examples and
intuitions I
x1

x1

Machine Learning

Andrew Ng

Simple example: AND 1.0

Example: OR function

-10

20 0 0
20 0 1
0 0
1 0
0 1 1 1
1 0
1 1

Andrew Ng Andrew Ng

5
 12/5/2023

Neural Networks:
Representation
Examples and
intuitions II
Machine Learning

Putting it together:

-30 10 -10

Negation: 20 -20 20

20 -20 20

0
1
0 0
0 1
1 0
1 1

Andrew Ng Andrew Ng

Neural Network intuition Handwritten digit classification

Layer 1 Layer 2 Layer 3 Layer 4

Andrew Ng [Courtesy of Yann LeCun] Andrew Ng

6
 12/5/2023

Handwritten digit classification

[Courtesy of Yann LeCun] Andrew Ng Andrew Ng

Multiple output units: One-vs-all.

Neural Networks:
Representation Pedestrian Car Motorcycle Truck

Multi-class
classification
Machine Learning Want , , , etc.
when pedestrian when car when motorcycle
Andrew Ng Andrew Ng

Multiple output units: One-vs-all.

Want , , , etc.
when pedestrian when car when motorcycle
Training set:

one of , , ,
pedestrian car motorcycle truck
Andrew Ng Andrew Ng