Assignment 4

Introduction to Machine Learning

Prof. B. Ravindran
1. Which of the following statements about Kernel Functions are TRUE? Assume, in each case,
that the vector x has 2 dimensions.
(a) The implicit vector transformation for the kernel K(x, x0 ) = (1+ < x, x0 >)4 has 9
dimensions.
(b) The implicit vector transformation for the kernel K(x, x0 ) = tanh(< x, x0 >) has ∞
dimensions.
(c) both (a) and (b)
(d) neither (a) nor (b)
Sol. This question can be answered in a straightforward manner using concepts from the
lecture ”SVM: Kernels”. Simple rewrite < x, x0 > as x1 .x01 + x2 .x02 , expand and group the
terms. The number of terms will be the numer of dimensions. Do not forget to include the
constant ’1’ as a dimension too.
(1+ < x, x0 >)4 has much more than 9 dimensions. tanh(< x, x0 >) is not a polynomial so it’s
unclear how to proceed. But every function can be expressed as it’s Taylor series summation
which implies an infinite polynomial. Thus, it has infinite dimensions. The intuition for this
is mentioned in the lecture ”SVM: Kernels”.
2. When using SVMs, what effect, in general, can you expect on the size of the margins when
the C parameter is decreased?
(a) the margins may become wider
(b) the margins may become narrower
(c) no relation between C and margin sizes
Sol. (a)

For Q3,4: Kindly download the synthetic dataset from the following link
https://bit.ly/2yXJJDA
The dataset contains 1000 points and each input point contains 3 features.
3. (2 marks) Train a linear regression model (without regularization) on the above dataset. Re-
port the coefficients of the best fit model. Report the coefficients in the following format:
β0 , β1 , β2 , β3
(a) -1.2, 2.1, 2.2, 1
(b) 1, 1.2, 2.1, 2.2
(c) -1, 1.2, 2.1, 2.2
(d) 1, -1.2, 2.1, 2.2
(e) 1, 1.2, -2.1, -2.2
Sol. (d)

1
4. (2 marks) Train a l2 regularized linear regression model on the above dataset. Vary the
regularization parameter from 1 to 10. As you increase the regularization parameter, absolute
value of the coefficients (excluding the intercept) of the model:
(a) increase
(b) first increase then decrease
(c) decrease
(d) first decrease then increase
(e) does not change
Sol. (c)

For Q5,6: Kindly download the modified version of Iris dataset from this link.
Available at: (https://goo.gl/vchhsd)
The dataset contains 150 points and each input point contains 4 features and belongs to one
among three classes. Use the first 100 points as the training data and the remaining 50 as test
data. (Note: Do not change the order of data points.)
5. (2 marks) Train a l2 regularized logistic regression classifier on the modified iris dataset. We
recommend using sklearn. Use only the first two features for your model. We encourage you
to explore the impact of varying different hyperparameters of the model. Kindly note that the
C parameter mentioned below is the inverse of the regularization parameter λ. As part of the
assignment train a model with the following hyperparameters:
Model: logistic regression with one-vs-rest classifier, C = 1e4
For the above set of hyperparameters, report the best classification accuracy
(a) 0.88
(b) 0.86
(c) 0.92
(d) 0.68
Sol. (b)
6. (2 marks) Train an SVM classifier on the modified iris dataset. We recommend using sklearn.
Use only the first two features for your model. We encourage you to explore the impact
of varying different hyperparameters of the model. Specifically try different kernels and the
associated hyperparameters. As part of the assignment train models with the following set of
hyperparameters
RBF-kernel, gamma = 0.5, one-vs-rest classifier, no-feature-normalization.
Try C = 0.01, 1, 10. For the above set of hyperparameters, report the best classification
accuracy along with total number of support vectors on the test data.
(a) 0.88, 69
(b) 0.44, 69
(c) 0.68, 44
(d) 0.34, 44
Sol. (a)