Multilayer Perceptron

1
 Nonlinear Input-Output Mapping

Approximation of functions

input nodes output nodes

m0-dimensional inner product space M-dimensional IP space

Continuously differentiable (like the activation functions)

Number of hidden layers

2
 Universal Approximation Theorem

Weights of a single Weights of a single Input to NN!

output neuron hidden layer
without nonlinearity
3
 • Generalization of Fourier Series? (finite, hence
approximation)

• A single hidden layer is sufficient for a multilayer

perceptron to compute an appropriate approximation
to a given training set represented by the set of input?!

• Does not say that a single hidden layer is optimum in

the sense of learning time, ease of implementation, or
generalization.

Bounds…

Function approximation on training data, Estimator on the

test data!

4
 Multi-dimensional Fourier Transform

Assuming integrability, first absolute moment of

Fourier magnitude distribution is defined

Euclidean

… bound on the approximation

5
 Number of neurons in the
single hidden layer

This comes after all the conditions of the universal

approximation theorem is met!

Another!
Input nodes

Total hidden neurons

6
 Curse of dimensionality
Minimization with respect to N

Curse of Rate of convergence

dimensionality to the minimization

Lesser availability of dense samples for the

estimation, while higher could be the function
complexity with increase in dimension!
7