Program: B.

Tech VII Semester

CSL0777: Machine Learning

Unit No. 2
Supervised learning Part-1

Lecture No. 17
Regularization in Linear Regression
Mr. Praveen
Gupta
Assistant Professor,
CSA/SOET
Outlines
• Regularization in Linear Regression
• How does Regularization Work?
• Techniques of Regularization
• Implementing L1 and regularization
using
L2 Python
• References
 Student Effective Learning Outcomes(SELO)
01: Ability to understand subject related concepts clearly along with
contemporary issues.
02: Ability to use updated tools, techniques and skills for effective domain
specific practices.
03: Understanding available tools and products and ability to use it effectively.
 Regularization in Linear Regression
•One of the major aspects of training your machine
learning model is avoiding overfitting. The model will
have a low accuracy if it is overfitting. This happens
because your model is trying too hard to capture the
noise in your training dataset.
•By noise we mean the data points that don’t really
represent the true properties of your data, but random
chance. Learning such data points, makes your model
more flexible, at the risk of overfitting.
•Regularization is one of the most important concepts of
machine learning. It is a technique to prevent the model
from overfitting by adding extra information to it.

4 / 22
Regularization in Linear Regression
•Sometimes the machine learning model performs
well with the training data but does not perform
well with the test data. It means the model is not
able to predict the output when deals with unseen
data by introducing noise in the output, and hence
the model is overfitted.
•Regularization technique can be used in such a way
that it will allow to maintain all variables or features
in the model by reducing the magnitude of the
variables. Hence, it maintains accuracy as well as a
generalization of the model.
4 / 22
How does Regularization Work?
•It mainly regularizes or reduces the coefficient of features
toward zero. In simple words, "In regularization technique,
we reduce the magnitude of the features by keeping the
same number of features."
•Regularization works by adding a penalty or complexity
term to the complex model. Let's consider the linear
regression equation:
Y ≈ β0 + β1X1 + β2X2 + …+ βpXp
•Here Y represents the learned relation and β represents the
coefficient estimates for different variables or predictors(X).
• Linear regression models try to optimize the intercept and
coefficients to minimize the cost function.

4 / 22
 How does Regularization Work?
• The equation for the cost function for the
linear model is given below:

•Now, we will add a loss function and

optimize parameter to make the model that
can predict the accurate value of Y. The loss
function for the linear regression is called
as RSS or Residual sum of squares.

4 / 22
 Techniques of Regularization

Ridge Regression:
•Ridge regression is one of the types of linear regression in
which a small amount of bias is introduced so that we can
get better long-term predictions.
•Ridge regression is a regularization technique, which is used
to reduce the complexity of the model. It is also called as L2
regularization.
•In this technique, the cost function is altered by adding the
penalty term to it. The amount of bias added to the model is
called Ridge Regression penalty. We can calculate it by
multiplying with the lambda to the squared weight of each
individual feature.
4 / 22
 Techniques of Regularization

• Ridge Regression:
•The equation for the cost function in ridge regression will
be:

•In the above equation, the penalty term regularizes the

coefficients of the model, and hence ridge regression
reduces the amplitudes of the coefficients that decreases
the complexity of the model.
4 / 22
 Techniques of Regularization

• Ridge Regression:
•As we can see from the above equation, if the values of λ
tend to zero, the equation becomes the cost function of the
linear regression model. Hence, for the minimum value of λ,
the model will resemble the linear regression model.
•A general linear or polynomial regression will fail if there is
high collinearity between the independent variables, so to
solve such problems, Ridge regression can be used.
•It helps to solve the problems if we have more parameters
than samples.

4 / 22
 Techniques of Regularization

Lasso Regression:
•Lasso regression is another regularization technique to
reduce the complexity of the model. It stands for Least
Absolute and Selection Operator.
•It is similar to the Ridge Regression except that the penalty
term contains only the absolute weights instead of a square
of weights.
• Since it takes absolute values, hence, it can shrink the slope
to 0, whereas Ridge Regression can only shrink it near to 0.
• It is also called as L1 regularization.

4 / 22
 Techniques of Regularization

• Lasso Regression:
•The equation for the cost function of Lasso regression will
be:

•Some of the features in this technique are completely

neglected for model evaluation.
• Hence, the Lasso regression can help us to reduce the
overfitting in the model as well as the feature selection.
4 / 22
Key Difference between Ridge Regression and
Lasso Regression
•Ridge regression is mostly used to reduce the overfitting in
the model, and it includes all the features present in the
model. It reduces the complexity of the model by shrinking
the coefficients.
• Lasso regression helps to reduce the overfitting in the
model as well as feature selection.

4 / 22
 Implementing L1 and L2 regularization
using Python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge,
Lasso
from sklearn.model_selection import train_test_split,
cross_val_score
from statistics import mean

4 / 22
 Implementing L1 and L2 regularization
using Python
# Loading and cleaning the Data
# Loading the data into a Pandas DataFrame
data = pd.read_csv('kc_house_data.csv')
# Dropping the numerically non-sensical variables
dropColumns = ['id', 'date', 'zipcode']
data = data.drop(dropColumns, axis = 1)
# Separating the dependent and independent variables
y = data['price']
X = data.drop('price', axis = 1)
# Dividing the data into training and testing set
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size = 0.25)
4 / 22
 Implementing L1 and L2 regularization
using Python
Step 3: Building and evaluating the different models #
Building and fitting the Linear Regression model
linearModel = LinearRegression()
linearModel.fit(X_train, y_train)

# Evaluating the Linear Regression model

print(linearModel.score(X_test, y_test))

4 / 22
 Implementing L1 and L2 regularization
using Python
# Ridge(L2) Regression:
# List to maintain the different cross-validation
scores cross_val_scores_ridge = []
# List to maintain the different values of
alpha alpha = []
# Loop to compute the different values of
cross-validation scores
for i in range(1, 9):
ridgeModel = Ridge(alpha = i *
0.25) ridgeModel.fit(X_train,
y_train)
scores = cross_val_score(ridgeModel, X, y, cv = 10)
avg_cross_val_score = mean(scores)*100
cross_val_scores_ridge.append(avg_cross_val_score)
4 / 22
alpha.append(i * 0.25)
 Implementing L1 and L2 regularization
using Python
# Ridge(L2) Regression:
# Loop to print the different values of cross-validation scores
for i in range(0, len(alpha)):
print(str(alpha[i])+' : '+str(cross_val_scores_ridge[i]))

From the above output, we can conclude that the best value
of alpha for the data is 2.
4 / 22
 Implementing L1 and L2 regularization
using Python
# Ridge(L2) Regression:
# Building and fitting the Ridge Regression model
ridgeModelChosen = Ridge(alpha = 2)
ridgeModelChosen.fit(X_train, y_train)

# Evaluating the Ridge Regression model

print(ridgeModelChosen.score(X_test, y_test))

4 / 22
 Implementing L1 and L2 regularization
using Python
# Lasso(L1) Regression: :
# List to maintain the cross-validation
scores cross_val_scores_lasso = []
# List to maintain the different values of
Lambda Lambda = []
# Loop to compute the cross-validation scores
for i in range(1, 9):
lassoModel = Lasso(alpha = i * 0.25, tol =
0.0925) lassoModel.fit(X_train, y_train)
scores = cross_val_score(lassoModel, X, y, cv = 10)
avg_cross_val_score = mean(scores)*100
cross_val_scores_lasso.append(avg_cross_val_score)
Lambda.append(i * 0.25)
4 / 22
 Implementing L1 and L2 regularization
using Python
# Lasso(L1) Regression:
# Loop to print the different values of cross-validation scores
for i in range(0, len(alpha)):
print(str(alpha[i])+' : '+str(cross_val_scores_lasso[i]))

From the above output, we can conclude that the best value
of lambda for the data is 2.
4 / 22
 Implementing L1 and L2 regularization
using Python
# Lasso(L1) Regression:
# Building and fitting the Lasso Regression Model
lassoModelChosen = Lasso(alpha = 2, tol = 0.0925)
lassoModelChosen.fit(X_train, y_train)

# Evaluating the Lasso Regression model

print(lassoModelChosen.score(X_test, y_test))

4 / 22
 Implementing L1 and L2 regularization
using Python
Step 4: Comparing and Visualizing the results
# Building the two lists for visualization
models = ['Linear Regression', 'Ridge Regression', 'Lasso Regression']
scores = [linearModel.score(X_test, y_test),
ridgeModelChosen.score(X_test, y_test),
lassoModelChosen.score(X_test,
y_test)]

# Building the dictionary to compare the scores

mapping = {}
mapping['Linear Regreesion'] = linearModel.score(X_test, y_test)
mapping['Ridge Regreesion'] = ridgeModelChosen.score(X_test, y_test)
mapping['Lasso Regression'] = lassoModelChosen.score(X_test, y_test)
# Printing the scores for different models
for key, val in mapping.items():
print(str(key)+' : '+str(val)) 4 / 22
 Implementing L1 and L2 regularization
using Python
Step 4: Comparing and Visualizing the results

4 / 22
 Learning Outcomes

The students have learn and understand the followings

• Regularization in Linear Regression
• How does Regularization Work?
• Techniques of Regularization
• Implementing L1 and L2 regularization using Python
 References

1. Machine Learning for Absolute Beginners by Oliver Theobald. 2019

2. http://noracook.io/Books/Python/introductiontomachinelearningwithpyth
on.pdf
3. https://www.tutorialspoint.com/machine_learning_with_python/machine
_learning_with_python_tutorial.pdf
Thank you