Supervised Learning

Multiple Linear Regression

Dr. Mamadou Camara(1)

(1) ESP, Cheikh Anta Diop University, Dakar, Senegal
mamadou.camara@ucad.edu.sn

Machine Learning course

Dr. Mamadou Camara. ML 1 / 11

I Multiple linear regression makes it possible to represent the
relationship of dependence between
I a continuous target variable
I and continuous predictors (Han and Kamber 2001).
I This relation is knowledge extracted from a base of examples
composed of n observations assumed to be independent.
I The linear model or linear equation can be written in matrix
form as follows (Draper and Smith 1998) (Cohen, Cohen et
al. 2003) :

Y = Xβ + ε (1)

Dr. Mamadou Camara. ML 2 / 11

I Y : the column vector of dimension n containing the
observations of the target variable, n representing the size of
the database of examples.
I X : the matrix of size n × (p + 1) composed of a column of 1s
followed by p columns of the observation vectors of the
explanatory variables.
I p represents the number of explanatory variables

Dr. Mamadou Camara. ML 3 / 11

I β : is the dimension (p+1) vector of the parameters to be
estimated.
I β0 = m is the y-intercept which represents the value of the
target variable Y when the explanatory variables are zero.
I The βj are the regression coefficients for estimating Y from X.
I Each βj represents the unit rate of change in Y per unit
change in the explanatory variable Xj .
I ε : n-dimensional vector of errors (standard error).

Dr. Mamadou Camara. ML 4 / 11

Dr. Mamadou Camara. ML 5 / 11
I The most used method to solve this equation is the method of
least squares.
I The principle of least squares consists in finding the values of
the parameters which minimize the sum of the squares of the
errors, namely :

n
X
S= ε2i (2)
i=1

Dr. Mamadou Camara. ML 6 / 11

I In general, there is no solution for the system Y = X β car il
est overdetermined, it has more equations than unknowns
(ceee 2007).
I An equivalent system of equations called "the normal
equations" can be used to find the solution (Cohen and Cohen
2003) :
I If X’X is invertible then these normal equations admit a
solution (Draper and Smith 1998 ; Cohen and Cohen 2003)
and 1 :

β = (X 0 X )−1 (X 0 Y ) (3)

1. X’ represents the transpose of X

Dr. Mamadou Camara. ML 7 / 11
I When the number of predictors is limited to 1, we have a
simple linear regression :

Y = βX + ε = β0 + β1 X1 + ε (4)

I The linear regression estimate is represented by a linear line

(Ŷ ).
I For each data point in the plot, an error ε is associated with
the distance between the point (Y) and the regression line (Ŷ )

Dr. Mamadou Camara. ML 8 / 11

Dr. Mamadou Camara. ML 9 / 11
TP

I DS 10 folds CV
I DS Multiple linear regression : calculation of Beta β
I DS Cross Validation and Multiple Linear Regression

Dr. Mamadou Camara. ML 10 / 11

Fin

Dr. Mamadou Camara. ML 11 / 11

References I