IE4213 Regression Tutorial

1) Suppose that the MSE of model A is less than the MSE of model B. What can we say about the R2
of models A and B?

2) Show that adding more factors increases the R2 of a linear regression model.

3) Suppose we have the weight and height of 5 individuals:

A B C D E
Weight (kg) 57.48 56.57 66.11 65.10 70.19
Height (m) 1.52 1.60 1.68 1.75 1.80

We are interested in modelling the relationship as follows:

𝑦𝑦 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥
where 𝑦𝑦 is weight (kg) and 𝑥𝑥 is height (m).

a) Express RSS of the above regression line as a function of 𝛽𝛽0 and 𝛽𝛽1 .
b) Find the stationary point of the above RSS function (i.e., 𝛽𝛽0 and 𝛽𝛽1 values such that
derivative of RSS equals zero).
c) Determine if the stationary point is a minimum, maximum or inflection point.

4) Revisit Question 3 using ridge regression.

a) Express loss of as a function of 𝛽𝛽0 and 𝛽𝛽1 . Assume 𝛼𝛼 = 1.
b) Find the stationary point of the above loss function. Assume 𝛼𝛼 = 1.
c) Express the stationary point of the above loss function as a function of 𝛼𝛼.
d) Does 𝑅𝑅 2 increase or decrease with 𝛼𝛼? Does this make sense?

5) Verify your solutions to Question 4 using scikit-learn Ridge.