Empherical Risk minimization

In words...
The Empirical Risk Minimization (ERM) principle is a learning paradigm which consists in selecting the
model with minimal average error over the training set. This so-called training error can be seen as an
estimate of the risk (due to the law of large numbers), hence the alternative name of empirical risk.

By minimizing the empirical risk, we hope to obtain a model with a low value of the risk. The larger the
training set size is, the closer to the true risk the empirical risk is.

If we were to apply the ERM principle without more care, we would end up learning by heart, which we
know is bad. This issue is more generally related to the overfitting phenomenon, which can be avoided by
restricting the space of possible models when searching for the one with minimal error. The most severe
and yet common restriction is encountered in the contexts of linear classification or linear regression.
Another approach consists in controlling the complexity of the model by regularization.