Regression: Matrix Formulation (philender.

com)

Table of Contents (bu.edu)

5.3 - The Multiple Linear Regression Model | STAT 462 (psu.edu)

5.4 - A Matrix Formulation of the Multiple Regression Model | STAT 462 (psu.edu)

https://support.minitab.com/en-us/minitab/21/help-and-how-to/statistical-modeling/
regression/supporting-topics/regression-models/what-is-the-standard-error-of-the-coefficient/ 
final step in MLR -CALC STD ERROR-TVAL-THEN TAKE PVAL

BEST MATERIAL FOR MLR

DOWN HERE

https://medium.com/@bhanu0925/deriving-normal-equation-for-multiple-linear-regression-
85241965ee3b

Search

Write

Get unlimited access to the best of Medium for less than $1/week.

Become a member
Deriving Normal Equation
for Multiple Linear
Regression

Bhanumathi Ramesh
·

Follow
4 min read

Oct 19, 2021

The mathematical formulation of Normal Equation

In this article, we will be seeing the steps to derive the normal
equation for multiple linear regression.

What is Multiple Linear Regression?

Multiple Linear Regression is a supervised machine learning

algorithm. This algorithm uses more than one independent variable
to predict a target variable. The below table is an example of a
multiple linear regression problem.

Table with 4 independent variables and 1 dependent variable

The above table has marks details and SAT score of 5 students. Each
subject is a independent variable( as called predictors,explanatory
variable or regressor (in regression problem)) and one dependent
variable (also called label,target,response variable). Since we have
more than one independent variable we have to find
the coefficients for each independent variable. In my Simple
Linear Regression article, we have discussed how to find the
coefficients. Now we will extent simple linear to multiple linear and
find the coefficients.

Mathematical Derivation

We know the equation of a line for simple linear regression is

Equation of a line

Similarly, now for the multiple linear regression, the formula will be,

Equation of a plane with multiple independent variables

The above formula can be rewritten in matrix form as,

Matrix representation for n records

This can be further rewritten as,

Now the above matrix can be represented as (all capital letters

represents Matix in further equations)

matrix representation

we know, the error is given by (Actual — Predicted), So the error can

be represented in matrix form as below.
we know, for linear regression, for n observations, the error/cost
function is given by,

Error/cost function

Lets, derive an equation for the error function to represent in matrix

form

Expanding the equation 1, we can rewrite it as,

Expanding the error function

The above equation can be rewritten in matrix form as,

Representing error function in matrix form

Hence, the Error/Cost function in matrix notation is given by,

Error Function in Matrix notation

Now the error equation becomes,

By Linear algebra,

Substituting the values in equation 2, equation 2 can be rewritten as,

Replacing, Yhat = Xβ in equation 3,

To simplify equation 4, we have to prove the following term as equal

solving equation 5,

after simplifying, equation 4 becomes,

Applying partial derivatives concerning β,

By applying matrix differentiation on equation 8,

simplifying the equation further,

After simplifying we have got a β coefficient matrix.

In the above equation X represents X_train, Y represents y_train

from the dataset. Plugging those values into the above equations will
give us the β coefficient matrix. This matrix will have (independent
variables + 1) different values. Suppose if your data set has 5
independent variables, your coefficients will be 6.

Predictions can be made using X_Test and β coefficient. That will be

the dot product of X_test and β coefficient.

Conclusion

This normal equation is used in Sci-kit learn’s LinearRegression()

which works well for small datasets. LinearRegression()’s
computational complexity is too high since it involves calculating
matrix inverse operation and complexity is o(n³). Please
refer complexity of matrix inverse operation. To overcome these
complexity issues, SGDRegressor() method is used, which uses
gradient descent to find the coefficients.

Please refer to the practical implementation of Normal Equation

here — https://git.io/J6Wes

LinkedIn: Bhanumathi Ramesh

Gradient Descents — Batch, Stochastic, and Minibatch

An act of moving downwards

3 min read·Nov 6, 2021

5
Bhanumathi Ramesh

Must know Regression Matrics of Machine Learning

What’s measured improves…

5 min read·Nov 11, 2021

See all from Bhanumathi Ramesh

Recommended from Medium

Sruthy Nath

Unraveling Multivariate Calculus in Machine Learning:

Gradients and Hessians
In the exciting world of machine learning, where complex models
harness the power of data to make predictions and decisions,
multivariate…

2 min read·Aug 9, 2023

1

Toby

95. Fourier Transform?

Today, I’m going to write a very interesting piece.

4 min read·Dec 23, 2023

105
Lists
Predictive Modeling w/ Python
20 stories·803 saves

Practical Guides to Machine Learning

10 stories·934 saves

Natural Language Processing

1105 stories·577 saves
ChatGPT
23 stories·403 saves

Technical Panchayat

Understanding Ridge Regression vs. Lasso

Regression: A Mathematical Comparison
Ridge and Lasso Regression are vital for handling multicollinearity
and feature selection in linear regression.
3 min read·Sep 27, 2023

Rafał Buczyński

in

Python in Plain English

Understanding Multiple Linear Regression in Machine
Learning
1. Introduction to Multiple Linear Regression

16 min read·Aug 4, 2023

Jaleel Adejumo

Regression Analysis from Scratch: Incorporating p-

value Computation
In a previous article, I explained the concept of gradient descent
and its application in a linear model. If you’d like to delve deeper…

7 min read·Aug 13, 2023

127

Everton Gomede, PhD

Step by Step of Principal Component Analysis

Introduction
7 min read·Nov 7, 2023

100

See more recommendations

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams