Multiple Regression

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 1

Regression analysis
• Regression analysis is a technique which is used to find a causal
relationship between dependent variable and independent variables.
• For example

• Relationship between effective time of study and marks scored

• Relationship between average food intake and weight

• Relationship between investment on marketing, product cost and demand of

product

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 2

• A brief classification of regression models
• Regression models can be classified based on –
➢No. of independent variables-
• If one independent variable, then it is known as simple regression model
• If two or more independent variable, then it is known as multiple regression model

➢Shape of regression curve-

• regression models can also be classified based on the shape of the curve; i.e. polynomial,
exponential, logarithmic etc.

➢Type of dependent (output) variable (y)-

• If dependent variable is continuous (output is a numeric value)- simple linear regression,
multiple linear regression, polynomial regression etc.
• If dependent variable is categorial (only binary output will come as 0 or 1, i.e. pass or fail,
Yes or No, Married or Unmarried, Profit or Loss, Win or Defeat etc.)- logistic regression,
Probit regression etc.
Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 3
Linear Regression Model
• A regression model is called “Linear” if
• The power of the regression coefficients (also called as Parameter) should be
maximum one.
• The first derivative w.r.t. any parameter should not have
• Each term in model is one of the following-
• A constant
• A parameter multiplied by an independent variable
• Equation of linear regression model is

• 𝑫𝒆𝒑𝒆𝒏𝒅𝒆𝒏𝒕 𝒗𝒂𝒓𝒊𝒂𝒃𝒍𝒆 = 𝒄𝒐𝒏𝒔𝒕𝒂𝒏𝒕 + 𝒑𝒂𝒓𝒂𝒎𝒆𝒕𝒆𝒓𝟏 ∗ 𝒊𝒏𝒅𝒆𝒑𝒆𝒏𝒅𝒆𝒏𝒕 𝒗𝒂𝒓𝒊𝒂𝒃𝒍𝒆𝟏 + ⋯ +

𝒑𝒂𝒓𝒂𝒎𝒆𝒕𝒆𝒓𝒏 ∗ 𝒊𝒏𝒅𝒆𝒑𝒆𝒏𝒅𝒆𝒏𝒕 𝒗𝒂𝒓𝒊𝒂𝒃𝒍𝒆𝒏

• Mathematically; 𝑦 = 0 + 1 𝑥1 + 2 𝑥2 +⋯+ 𝑚 𝑥𝑚

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 4

Non-linear Regression Model
• Simply, it doesn’t follow the rules of linear model
• A regression model is called “non-linear” if the derivative w.r.t. parameter
depends on one or more parameters.
𝑫𝒆𝒑𝒆𝒏𝒅𝒆𝒏𝒕 𝒗𝒂𝒓𝒊𝒂𝒃𝒍𝒆
𝒌
= 𝒄𝒐𝒏𝒔𝒕𝒂𝒏𝒕 + 𝒑𝒂𝒓𝒂𝒎𝒆𝒕𝒆𝒓𝟏 ∗ 𝒊𝒏𝒅𝒆𝒑𝒆𝒏𝒅𝒆𝒏𝒕 𝒗𝒂𝒓𝒊𝒂𝒃𝒍𝒆𝟏 + ⋯ + 𝒑𝒂𝒓𝒂𝒎𝒆𝒕𝒆𝒓𝒏
∗ 𝒊𝒏𝒅𝒆𝒑𝒆𝒏𝒅𝒆𝒏𝒕 𝒗𝒂𝒓𝒊𝒂𝒃𝒍𝒆𝒏

𝝏𝒀
Linear model: 𝑌 = 𝑎 + 𝑏𝑋 = 𝑋;
𝝏𝒃

𝝏𝒀 𝝏𝒀
Linear model: 𝑌 = 𝑎 + 𝑏𝑋 + 𝑐𝑋 2 = 𝑋; = 𝑋2
𝝏𝒃 𝝏𝒄

𝝏𝒀 𝝏𝒀
Non-Linear model: 𝑌 = 𝑎 + 𝑏 2 𝑋 + 𝑐𝑋 = 2𝒃𝑋; =𝑋
𝝏𝒃 𝝏𝒄

𝝏𝒀
Non-Linear model: 𝑌 = 𝑎 + (ln 𝑏) 𝑋 ; it is non-linear because 𝝏𝒃 will depend on b.

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 5

Linear regression: dependent/ output variable (y) is continuous in nature;
i.e. it takes numerical values.

Simple linear regression equation Multiple linear regression equation

• Linear with Only one independent • Linear with More than one
variable (x) and one dependent independent variables (x) and one
variable (y) dependent variable (y)
• Equation 𝑦= 0 + 1𝑥 +𝜀 •𝑦= 0+ 1 𝑥1 + 2 𝑥2 + ⋯+
𝑚 𝑥𝑚
where:
𝒊 is coefficient of regression (also known as parameter) model
𝜺 is a random variable called the error term

NOTE- For better explanation in Hindi, I recommend you to watch this 15 minute video on Youtube
https://www.youtube.com/watch?v=lzGKRSvs5HM copy this link to your browser

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 6

• Multiple Linear regression eq. 𝑦 = 0 + 1 𝑥1 + 2 𝑥2 + ⋯+ 𝑚 𝑥𝑚
• Example :-
weight = 𝑎 + 1 ∗ food intake + 2 ∗ 𝑒𝑥𝑒𝑟𝑐𝑖𝑠𝑒 𝑡𝑖𝑚𝑒 + 3 ∗ 𝑖𝑛𝑐𝑜𝑚𝑒
• Since, weight of a person is majorly depends upon food taken by person and time
spent in physical exercise. It doesn't or very less likely to depend on the income of
a person.

• So, effect of factors of food intake and exercise time is much higher than effect of
income on body weight of a person.
• i.e. independent variables food intake and exercise time best explains the
dependent variable weight
• So, parameters 1 , 2 must be much higher than 3
Mathematically ( 1 , 2 >>> 3)

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 7

Stepwise Regression
Suppose a situation where an Dependent Variable (y) depends upon
various Independent Variables. Some of them significantly affect the
output (D.V.) whereas some I.V. doesn’t affect as much as others.
• 𝑦 = 0 + 1 𝑥1 + 2 𝑥2 + 3 𝑥3 + 4 𝑥4 + 5 𝑥5 + 6 𝑥6 + 7 𝑥7 +
8 𝑥8 + 9 𝑥9 + 10 𝑥10 + 11 𝑥11 + 12 𝑥12 +
• In stepwise regression, we try to make/fit a equation (regression
model) which contains only those independent variables who affects
the output significantly.
• In stepwise regression, a variable is selected for either addition or
subtraction from set of variables
Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 8
Stepwise Regression cont…
So, the main objective of Stepwise Regression is to choose a subset of
the independent variables which “best” explains the dependent
variable.
➢ There are 3 main approaches to carry out a stepwise regression

• Forward Selection

• Backward Elimination

• Bi-directional Elimination

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 9

Forward selection
• Starting with no variables in the model
• Perform a testing of addition of each variable one by one using pre-
defined criteria
• Add the variable whose inclusion/addition gives the most significant
improvement to the model
• Repeat the process until no significant improvement will obtain to the
model
• Now model contains only those variable who have significant impact
on the output.

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 10

Backward Elimination
• Start with all the variables in the model
• Perform a testing of deletion of each variable one by one using pre-
defined criteria
• Drop the “least significant” variable one at a time
• Repeat the process until deletion of variable doesn’t affect the model
• Now model contains only those variable who have significant impact
on the output.

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 11

Bi-directional elimination

• It is a combination of both forward selection and backward

elimination
• Perform the testing at each step for variables to be included or
dropped.
• Perform a forward selection, but drop the variable which become no
longer significant after introduction of new variables.

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 12

Lecture-06, Course- 4ME2-01, IV Sem (Mech. Engg.) 13