Assignment 3

Introduction to Machine Learning

Prof. B. Ravindran
1. Consider the case where two classes follow Gaussian distribution which are centered at (6, 8)
and (−6, −4) and have identity covariance matrix. Which of the following is the separating
decision boundary using LDA assuming the priors to be equal?
(a) x + y = 2
(b) y − x = 2
(c) x = y
(d) both (a) and (b)
(e) None of the above
(f) Can not be found from the given information
Sol. (a)
As the distribution is Gaussian and has identity covariance (which are equal), the separating
boundary will be linear. The decision boundary will be orthogonal to the line joining the
centers and will pass from the midpoint of centers.

2. Which of the following are differences between PCR and LDA?

(a) PCR is unsupervised whereas LDA is supervised
(b) PCR maximizes the variance in the data whereas LDA maximizes the separation between
the classes
(c) both (a) and (b)
(d) None of these
Sol. (c)
Refer to the lecture.
3. Which of the following are differences between LDA and Logistic Regression?
(a) Logistic Regression is typically suited for binary classification, whereas LDA is directly
applicable to multi-class problems
(b) Logistic Regression is robust to outliers whereas LDA is sensitive to outliers
(c) both (a) and (b)
(d) None of these
Sol. (c)
Logistic regression uses the sigmoid function, and the output values are between 0 and 1, so it
is typically suited for binary classification. LDA can be used when there are multiple classes
present.
In Logistic Regression, the effect of outliers is dampened because of sigmoid transformation. In
LDA, the objective function is based on the distance, which can change drastically if outliers
are present.

1
4. We have two classes in our dataset. The two classes have the same mean but different
variance.
(a) LDA can classify them perfectly.
(b) LDA can NOT classify them perfectly.
(c) LDA is not applicable in data with these properties
(d) Insufficient information
Sol. (b) If the classes have the same mean, they will not be linearly separable.

5. We have two classes in our dataset. The two classes have the same variance but different
mean.
(a) LDA can classify them perfectly.
(b) LDA can NOT classify them perfectly.
(c) LDA is not applicable in data with these properties
(d) Insufficient information
Sol. (d) Depending on the actual values of the mean and variance, the two classes may or
may not be linearly separable.

6. Which of these techniques do we use to optimise Logistic Regression:

(a) Least Square Error
(b) Maximum Likelihood
(c) (a) or (b) are equally good
(d) (a) and (b) perform very poorly, so we generally avoid using Logistic Regression
(e) None of these
Sol. (b) Refer the lecture.
7. Suppose we have two variables, X and Y (the dependent variable), and we wish to find their
relation. An expert tells us that relation between the two has the form Y = meX + c. Suppose
the samples of the variables X and Y are available to us. Is it possible to apply linear regression
to this data to estimate the values of m and c?

(a) no
(b) yes
(c) insufficient information

Sol. (b)

Instead of considering the dependent variable directly, we can transform the independent
variable by considering the exponent of each value. Thus, on the X-axis, we can plot values
of eX , and on the Y-axis, we can plot values of Y . Since, the relation between the dependent
and the transformed independent variable is linear, the value of slope and intercept can be
estimated using linear regression.

2
 8. What might happen to our logistic regression model if the number of features is more than
the number of samples in our dataset?

(a) It will remain unaffected

(b) It will not find a hyperplane as the decision boundary
(c) It will overfit
(d) None of the above

Sol. (c)
Refer to the lecture.
9. Logistic regression also has an application in

(a) Regression problems

(b) Sensitivity analysis
(c) Both (a) and (b)
(d) None of the above

Sol. (b)
Refer to the lecture.
10. Consider the following datasets:

Feature 1 Feature 2 Class

0 0 A
0 1 A
1 0 A
1 1 B

Dataset 1

Feature 1 Feature 2 Class

0 0 A
0 1 B
1 0 B
1 1 A

Dataset 2

Which of these datasets can you achieve zero training error using Logistic Regression (without
any additional feature transformations)?

(a) Both the datasets

(b) Only on dataset 1
(c) Only on dataset 2

3
(d) None of the datasets

Sol. (b)

Dataset 1 is linearly separable, so we will get zero training error when using logistic regression.
Dataset 2 is not linearly separable, so we will not get zero training error when using logistic
regression.