INDIAN INSTITUTE OF TECHNOLOGY, KHARAGPUR

Department of Industrial Engineering and Management

Class Test 1

Subject Number: IM31202 Subject Name: Statistical Learning with Applications

Full Marks: 50 Time: 1 hour Date: 02.02.2023

Instructions : 1. Attempt all questions.

2. Maximum marks are shown against each question.
3. Answers should be short and to the point.
4. Statistical and other tables as required are attached with the question
paper.
5. Assume 𝜶 = 𝟎. 𝟎𝟓 wherever applicable if not mentioned otherwise

1. Consider fitting a linear regression model, from the data below. 𝑋1 , 𝑋2 are features and 𝑌 is
the response. The relevant matrices are given below:

y x1 x2
193 7.2 4.0
230 8.2 5.3
172 6.2 3.9
91 2.7 5.2
113 6.6 10.7
125 5.0 5.7

(𝑿𝑻 𝑿)−𝟏

3.041 -0.316 -0.169

-0.316 0.055 -0.002
-0.169 -0.002 0.032

𝑿𝑻 𝒀
924
5960.531
5055.35

Hat Matrix
0.358 0.350 0.299 -0.022 -0.087 0.103
0.350 0.445 0.238 -0.217 0.137 0.047
0.299 0.238 0.287 0.144 -0.120 0.153
-0.022 -0.217 0.144 0.752 0.000 0.343
-0.087 0.137 -0.120 0.000 0.938 0.133
0.103 0.047 0.153 0.343 0.133 0.220

1
a. With the above data, fit a linear regression model and estimate the coefficients. (5)

𝜷 = (𝑿𝑻 𝑿)−𝟏 𝑿𝑻 𝒚 =

× =

b. State the hypothesis and perform ANOVA on this regression model, and interpret the
result. (10)

𝑯𝟎 : 𝜷 𝟏 = 𝜷 𝟐 = 𝟎
𝑯𝟏 : 𝒂𝒕 𝒍𝒆𝒂𝒔𝒕 𝒐𝒏𝒆 𝜷𝒋 ≠ 𝟎

NOTE: RESULTS OF ANOVA SHOULD BE PRESENTED IN THE TABULAR FORM

AS ABOVE

c. Perform hypothesis tests on the coefficients and interpret the importance of the
features. (10)

̂ 𝟐 𝑪𝒋𝒋
𝒔𝒆(𝜷𝒋 ) = √𝝈

2
̂ 𝟐 is estimated by MSE from the ANOVA table
𝝈
̂𝒋
𝑪𝒋𝒋 is the diagonal component of (𝑿𝑻 𝑿)−𝟏 matrix corresponding to 𝜷
𝛽 ̂
𝛽 24.16
for 𝛽1 : 𝑡0 1 = 𝑠𝑒(𝛽1 ) = = 7.31
1 √195.08∗0.056

𝛽 ̂
𝛽 −10.7
for 𝛽2 : 𝑡0 2 = 𝑠𝑒(𝛽2 ) = = −4.28
2 √195.08∗0.032

𝑡𝑐𝑟𝑖𝑡 = 𝑡0.975,3 = 3.182

𝛽 𝛽
Since both |𝑡0 1 | and |𝑡0 2 | are greater than 𝑡𝑐𝑟𝑖𝑡 , hence both the features are significant.
d. Calculate 𝑹𝟐 and 𝑹𝟐𝒂𝒅𝒋 and comment on the implications. (5)
𝑺𝑺𝑹
𝑹𝟐 = = 𝟎. 𝟗𝟔
𝑺𝑺𝑻
𝒏−𝟏 𝑺𝑺𝑬 𝟓 𝟓𝟖𝟓.𝟐𝟒
𝑹𝟐𝒂𝒅𝒋 = 𝟏 − (𝒏−𝒌−𝟏) (𝑺𝑺𝑻) = 𝟏 − (𝟑) ∗ ( 𝟏𝟒𝟏𝟏𝟐 ) = 𝟎. 𝟗𝟑

e. What is the average leverage of this model? From the leverage values, determine which
points have more influence on the model? (5)

Ans: Average leverage = (Trace of H)/n = 0.5

Leverage > 2p/n indicates influence
2p/n = 4/6 = 0.67
From the Hat matrix, 4th and 5th diagonal elements are greater than 0.67. Hence 4th and 5th
observations have more influence.

f. Calculate Cook’s distance for the 5th observation. Interpret the effect of this
observation. (5)
Residual of 5th observation
𝑟5 = 𝑦5 − 𝑦̂5 = 125 − (71.33 + 24.16 ∗ 6.6 − 10.7 ∗ 10.7) = 113 − 116.61 = −3.61
𝑟52 ℎ55 (−3.61^2)∗0.938
Cook’s distance = 𝐷5 = = = 98.08
𝑝(1−ℎ55 ) 2∗(1−0.938)

F_Crit = 𝐹0.5 (𝑝, 𝑛 − 𝑝) = 𝐹0.95,2,3 = 9.55

Since 𝐷5 ≫ 𝐹𝑐𝑟𝑖𝑡 it is evident that the 5th observation is significantly influencing the linear
model.

2. Explain the relation between Bias-variance trade-off and model flexibility. (5)

3
3. Explain KNN method in the context of numerical response and categorical response.
You may use a diagrams to explain the working principle. (5)

Hint:
For ANOVA of Regression
𝟐 𝟐
(∑𝒏
𝒊=𝟏 𝒚𝒊 ) (∑𝒏
𝒊=𝟏 𝒚𝒊 )
̂ ′ 𝑿′ 𝒚 −
𝑺𝑺𝑹 = 𝜷 ; 𝑺𝑺𝑻 = 𝒚′𝒚 −
𝒏 𝒏