AI Foundations and Applications

4. Linear Regression

Thien Huynh-The
HCM City Univ. Technology and Education
Jan, 2023
Course Contents

1. Introduction (1 week)
2. Intelligent Agents (1 week)
3. Machine Learning: Fundamental Concepts (3 weeks)
1. Feature Engineering
2. Linear Regression
3. Overfitting
4. Machine Learning Algorithms (3 weeks)
1. K-nearest neighbors Note: This is the tentative contents and may
2. K-means clustering be changed depending upon the teaching
3. Naive Bayes classifier progress.

5. Neural Networks (2 weeks) Note: There are no final exam of this course,
6. Deep Learning (4 weeks) therefore it has one reserved week for
marking
1. Convolutional Neural Networks
2. Applications with codes

HCMUTE AI Foundations and Applications 04/05/2023 2

Course Information

• The side will be delivered to students:

• Check website often (https://sites.google.com/site/thienhuynhthe/teaching)
• As recommend to print as hard copy
• Assignments and announcements will be posted on the website
• Requirements:
• PC or laptop (in-class coding with python or matlab)
• Course work
• Attendence: 10%
• In-class assigments: 30%
• Around 6 sudden assignments (5% of each)
• Students have prepared A4 papers (not allow others else)
• Homework programming assigment: 20% (individual)
• Final programming project: 40% (2 students per group)
• Please note: I do not accept late assigment submission unless any reasons.

HCMUTE AI Foundations and Applications 04/05/2023 3

Course Information (cont.)

• Homework assignment:
• Send code to teacher by email before deadline
• Over deadline is zero (no negotiate)
• Code with detailed comments (to rate how much students understand their codes)
• Run codes and record the laptop screen to demonstrate your codes (even it may be error)
• Final project:
• Send code to teacher by email before deadline
• Over deadline is zero (no negotiate)
• Code with detailed comments (to rate how much students understand their codes)
• Run codes directly on student’s laptop and show to teacher
• Answer question from teacher individually

HCMUTE AI Foundations and Applications 04/05/2023 4

Plagiarism

• All assignments are to be done individually.

• You can discuss the assignments with other students, but you should not give
your code (or parts of your code) to other students. You should not look at
another student’s code until after you have handed in your assignment (and
the due date is past).
• Plagiarism has occurred in the past in this class and it has had very negative
consequences for the students involved.
• Because 60% of the course mark is based on handed in work, I will be very
diligent about detecting plagiarism.
• Email policy:
• Do not answer questions about course content and queries about your marks.
• Do not need to ask me when you want to absent oneself from school.

HCMUTE AI Foundations and Applications 04/05/2023 5

What is Linear Regression ?

• Linear Regression is a machine learning algorithm based on supervised

learning. It performs a regression task
• Linear regression is an algorithm that provides a linear relationship between an
independent variable and a dependent variable to predict the outcome of
future events.
• It is a statistical method used in data science and machine learning for
predictive analysis.
• The independent variable is also the predictor or explanatory variable that
remains unchanged due to the change in other variables

HCMUTE AI Foundations and Applications 04/05/2023 6

What is Linear Regression ?

• There are many names for a regression’s dependent variable. It may be called
an outcome variable, criterion variable, endogenous variable, or regressand.
• The independent variables can be called exogenous variables, predictor
variables, or regressors.
• In regression set of records are presented with X and Y values and this values
are used to learn a function, so that if you want to predict Y from an unknown X
this learn function can be used.
• In regression we have to find value of Y, therefore, it is necessary to learn a
function in this case

HCMUTE AI Foundations and Applications 04/05/2023 7

What is Linear Regression ?

• A sloped straight line represents the linear

regression model.
• X-axis = Independent variable
• Y-axis = Output / dependent variable
• Line of regression = Best fit line for a model
• Here, a line is plotted for the given data
points that suitably fit all the issues. Hence, it
is called the ‘best fit line.’ The goal of the
linear regression algorithm is to find this best
fit line seen in the above figure.

HCMUTE AI Foundations and Applications 04/05/2023 8

Benefits of Linear Regression

• Easy implementation
• The linear regression model is computationally simple to implement as it does not demand
a lot of engineering overheads, neither before the model launch nor during its
maintenance.

• Interpretability
• Unlike other deep learning models (neural networks), linear regression is relatively
straightforward. As a result, this algorithm stands ahead of black-box models that fall short
in justifying which input variable causes the output variable to change.

HCMUTE AI Foundations and Applications 04/05/2023 9

Benefits of Linear Regression

• Scalability
• Linear regression is not computationally heavy and, therefore, fits well in cases where
scaling is essential. For example, the model can scale well regarding increased data
volume (big data).

• Optimal for online settings

• The ease of computation of these algorithms allows them to be used in online settings.
The model can be trained and retrained with each new example to generate predictions in
real-time.

HCMUTE AI Foundations and Applications 04/05/2023 10

Linear Regression Equation

• Let’s consider a dataset that covers RAM sizes ram capacity cost
and their corresponding costs. 2 12
• If we plot RAM on the X-axis and its cost on 4 17
the Y-axis, a line from the lower-left corner of 8 31
the graph to the upper right represents the 16 68
relationship between X and Y. cost
• Mathematically these slant lines follow the 80

following equation
70

60

y = m*X + b 50

40
• Where X: dependent variable (target) 30

• Y: independed varible 20

• m: slope of the line 10

0
0 2 4 6 8 10 12 14 16 18

HCMUTE AI Foundations and Applications 04/05/2023 11

Linear Regression Equation
ram capacity cost
• However, machine learning experts have a new 2 12
window experts have a different notation to the 4 17
above slope-line equation. 8 31
y(x) = p0 + p1*x 16 68
where cost
• y = output variable. Variable y represents the 80

continuous value that the model tries to predict. 70

• x = input variable. In ML, x is the feature, while it is 60

50
termed the independent variable in statistics. Variable 40
x represents the input information provided to the 30
model at any given time. 20
• p0 = y-axis intercept (or the bias term). 10

• p1 = the regression coefficient or scale factor. In 0

0 2 4 6 8 10 12 14 16 18
classical statistics, p1 is the equivalent of the slope of
the best-fit straight line of the linear regression model.
regression modeling is all about finding the values for the unknown
• pi = weights (in general). parameters of the equation, i.e., values for p0 and p1 (weights)

HCMUTE AI Foundations and Applications 04/05/2023 12

Multiple Linear Regression

• A regression model can be used for multiple features by extending the

equation for the number of variables available within the dataset.
• The formula for multiple linear regression would look like
y(x) = p0 + p1x1 + p2x2 + ... p(n)x(n)
• The machine learning model uses the above formula and different weight
values to draw lines to fit. Moreover, to determine the line best fits the data, the
model evaluates different weight combinations that best fit the data and
establishes a strong relationship between the variables.
• The regression model uses a cost function to optimize the weights (pi). The
cost function of linear regression is the root mean squared error or mean
squared error (MSE).

HCMUTE AI Foundations and Applications 04/05/2023 13

Multiple Linear Regression

• Given the simple linear equation y=mx+b, we

can calculate the MSE values:

where
• N: total number of observation (data points)
• yi: actual value of an observation
• mxi + b = prediction
• Along with the cost function, a ‘Gradient
Descent’ algorithm is used to minimize MSE
and find the best-fit line for a given training
dataset in fewer iterations

HCMUTE AI Foundations and Applications 04/05/2023 14

Types of Linear Regression

• Test: Describe all types of linear regression?

• Simple linear regression
• Multiple linear regression
• Logistic regression
• Ordinal regression
• Multinomial logistic regression

HCMUTE AI Foundations and Applications 04/05/2023 15

Example on Python

• Predict weight based on the human height

Height Weight
147 49
150 50
153 51
155 52
158 54
160 56
163 58
165 59
168 60
170 72
173 63
175 64
178 66
180 67
183 68

HCMUTE AI Foundations and Applications 04/05/2023 16

How to solve this problem?

• A function to describe the relation between input and output

where is the vector of weights (or model parameter) that we need to determine.
is the ground truth and is the predicted output. is the bias
• With regression, we expect the error between and is minimum, that means

• Loss function

Clearly, we expect the value of loss is minimum as to achieve this by minimizing

the loss function for

HCMUTE AI Foundations and Applications 04/05/2023 17

How to solve this problem?

• This function can be written shortly by matrix, vector, and norm as follows

where and
• The loss function should have derivative
• Find the optimal can be achieved by solving the derivative function

HCMUTE AI Foundations and Applications 04/05/2023 18

How to solve this problem?

• Derivative of the loss function of linear regression

• Calculate derivative function as zero

• If the function is reversible, we got

• If the function is non-reversible, the function has either no root or many root of
equation

HCMUTE AI Foundations and Applications 04/05/2023 19

How to solve this problem?

• With every matrix , there just exists one value that has norm as minimizing
, where is is pseudo inverse of
• The pseudo inverse of a matrix always exists
• When the matrix is squared and reversible, pseudo inverse is inverse in this
case
w

HCMUTE AI Foundations and Applications 04/05/2023 20

Python Code
from __future__ import division, print_function, unicode_literals
import numpy as np
import matplotlib.pyplot as plt

# height (cm)
X = np.array([[147, 150, 153, 158, 163, 165, 168, 170, 173, 175, 178, 180, 183]]).T
# weight (kg)
y = np.array([[ 49, 50, 51, 54, 58, 59, 60, 62, 63, 64, 66, 67, 68]]).T
# Visualize data
plt.plot(X, y, 'ro')
plt.axis([140, 190, 45, 75])
plt.xlabel('Height (cm)')
plt.ylabel('Weight (kg)')
plt.show()

AI Foundations and Applications

Python Code
# Building Xbar # Drawing the fitting line
one = np.ones((X.shape[0], 1)) plt.plot(X.T, y.T, 'ro') # data
Xbar = np.concatenate((one, X), axis = 1) plt.plot(x0, y0) # the fitting line
# Calculating weights of the fitting line plt.axis([140, 190, 45, 75])
A = np.dot(Xbar.T, Xbar)
b = np.dot(Xbar.T, y) plt.xlabel('Height (cm)')
w = np.dot(np.linalg.pinv(A), b) plt.ylabel('Weight (kg)')
print('w = ', w) plt.show()
# Preparing the fitting line
w_0 = w[0][0]
w_1 = w[1][0]
x0 = np.linspace(145, 185, 2)
y0 = w_0 + w_1*x0

AI Foundations and Applications

Python Code

Now using the model to predict the weights of two persons having heights of 155 and 160 cm.
The obtained function as y = 0.5592*x - 33.7354

HCMUTE AI Foundations and Applications 04/05/2023 23

Python Code
y1 = w_1*155 + w_0
y2 = w_1*160 + w_0
# y = -33.7354*x + 0.5592
print( u'Predict weight of person with height 155 cm: %.2f (kg), real number: 52 (kg)' %(y1) )
print( u'Predict weight of person with height 160 cm: %.2f (kg), real number: 56 (kg)' %(y2) )

HCMUTE AI Foundations and Applications 04/05/2023 24

Python Code

• Using the library in scikit-learn to determine w0 and w1

from sklearn import datasets, linear_model

# fit the model by Linear Regression

regr = linear_model.LinearRegression(fit_intercept=False) #
fit_intercept = False for calculating the bias
regr.fit(Xbar, y)

# Compare two results

print( 'Solution found by scikit-learn : ', regr.coef_ )
print( 'Solution found by (5): ', w.T)

HCMUTE AI Foundations and Applications 04/05/2023 25

HCMUTE AI Foundations and Applications 04/05/2023 26
HCMUTE AI Foundations and Applications 04/05/2023 27
HCMUTE AI Foundations and Applications 04/05/2023 28
HCMUTE AI Foundations and Applications 04/05/2023 29
HCMUTE AI Foundations and Applications 04/05/2023 30
HCMUTE AI Foundations and Applications 04/05/2023 31