LINEAR REGRESSION

MATHEMATICAL FOUNDATION:
Linear regression finds the best-fitting straight line (called the regression line) through a set
of points by minimizing the sum of squared errors between predicted and actual [Link]
mathematical formula is: y = mx + b where y is the dependent variable (prediction), x is the
independent variable (input feature), m is the slope (coefficient/weight),b is the y-intercept
(bias term)

To find optimal values for m and b, we use the Ordinary Least Squares (OLS) method which
minimizes:

min ∑(yi − (mxi + b))2

​ ​

This minimization can be solved by taking partial derivatives with respect to m and b and
setting them to zero, resulting in the optimal parameters for the regression line.

PRACTICAL EXAMPLE: Christmas Tree Price Prediction.

Given dataset:

Using linear To find m and b, we use the formulas:

Height (ft) Price (₹)
regression: y =
4 2,905 (derived from OLS, give us the optimal values
bx+ a
of m and b that minimize the prediction errors)
5 3,735
Where:
6 4,565 a = (∑ y − b ∑ x)/n
x = height
7 5,395
(independent
variable)
b = (n ∑ xy − ∑ x ∑ y)/(n ∑ x2 − (∑

y = price
(dependent
variable)

b = price
increase per
foot

a = base price

n=4 ∑x = 22 ∑y = 16,600 ∑xy = 95,450 ∑x² = 126

LINEAR REGRESSION 1
 Now, b = 830 and a = -415

Therefore, the linear regression equation is:

P rice = 830 × Height + 415

To predict the price of an 8-foot tree,

Substituting x = 8,

y = 830(8) + 415 = 6225

Therefore the predicted price for an 8-foot

tree would be ₹6225.

Source - Graph Pad

Source - Towards AI,Medium

DERIVATION :
Let there be n inputs and outputs of the form (xi , yi ) ​ ​

^i
The “line of best fit” is defined as Y ​
= a + BXi  ​

The error function to Now replacing Y^i with the ​ To minimise S, one needs to
be minimised, let’s regression equation, find where the first
say is S. derivative of S with respect
n
S = ∑(Yi − a − BXi )2
to a and b must be = 0.
n
S = ∑(Yi − Y^i )2
​ ​ ​

​ ​ ​
i=1
i=1

[Link] with respect to a,

n n n
∂S Yi − a − BXi = u
[∑(Yi − a − BXi )2 ] S = ∑ u2 0 = −2 ∑(Yi − a − BX
​ ​

∂a i=1
​ ​ ​ ​ ​ ​ ​

∂S i=1 i=1
= 2u
∂u
​

∂S
=
∂S ∂u
∗
∂u
= −1 a = Yˉ − Bx
ˉ
∂a ∂u ∂a ∂a
​ ​ ​ ​

[Link] with respect to b,

LINEAR REGRESSION 2
 ∂S
n ∂S ∂S ∂u ∂S ∂u
[∑(Yi − a − Bxi )2 ] ∂b
​
= ∗
∂u ∂b
​ ​

∂u
​
= 2u
∂b
​
= −x
∂b
​ ​ ​ ​

i=1
∑i=1 (xi Yi − Yˉ xi )
n
n B=
​ ​ ​ ​

0 = ∑ xi (Yi − a − Bxi )
n
∑i=1 (x2i − x
ˉxi )
​

n ∑(xy) − ∑ x ∗ ∑ y
​ ​ ​

=
​ ​ ​ ​
​

B
i=1 ∑(x2 − (∑ x)2 )
​

n
∑i=1 xi ​

ˉ=
​ ​

n x ​

n
0 = ∑ xi Yi − xi (Yˉ − Bx
​ ​ ˉ) − Bx2i )
​ ​ ​

n
i=1
ˉ ∑i=1 yi
Y =
​ ​

LINEAR REGRESSION 3