Advanced Machine Learning

Loss Function and Regularization

Amit Sethi
Electrical Engineering, IIT Bombay
 Learning outcomes for the lecture

• Write expressions for common loss functions

• Match loss functions to qualitative objectives

• List advantages and disadvantages of loss

functions
 Contents
• Revisiting MSE and L2 regularization

• How L1 regularization lead to sparsity

• Other losses inspired by L1 and L2

• Hinge loss leads to small number of support vectors

• Link between logistic regression and cross entropy

 Assumptions behind MSE loss
• MSE is related to RMSE

• RMSE is the standard deviation of the error

• The mean of the error will be zero for a convex

problem
 Regularization in regression
• Why regularize?
– Reduce variance, at the cost of bias
– Increase test (validation) accuracy
– Get interpretable models

• How to regularize?
– Shrink coefficients
– Reduce features
Regularization is constraining a model
• How to regularize?
– Reduce the number of parameters
• Share weights in structure
– Constrain parameters to be small
– Encourage sparsity of output in loss
• Most commonly Tikhonov (or L2, or ridge)
regularization (a.k.a. weight decay)
– Penalty on sums of squares of individual weights
𝑁 𝑛 𝑛
1 2
𝜆
𝐽= 𝑦𝑖 − 𝑓 𝑥𝑖 + 𝑤𝑗 2 ; 𝑓 𝑥𝑖 = 𝑤𝑗 𝑥𝑖 𝑗 ;
𝑁 2
𝑖=1 𝑗=1 𝑗=0
 Coefficient shrinkage using ridge

Source: Regression Shrinkage and Selection via the Lasso, by Robert Tibshirani, Journal of Royal Stat. Soc., 1996
L2-regularization visualized
 Contents
• Revisiting MSE and L2 regularization

• How L1 regularization lead to sparsity

• Other losses inspired by L1 and L2

• Hinge loss leads to small number of support vectors

• Link between logistic regression and cross entropy

 Subset selection
• Set the coefficients with lowest absolute value
to zero
 Level sets of Lq norm of coefficients

Which one is ridge? Subset selection? Lasso?

Source: Regression Shrinkage and Selection via the Lasso, by Robert Tibshirani, Journal of Royal Stat. Soc., 1996
 Other forms of regularization
• L1-regularization
(sparsity
inducing norm)
– Penalty on sums
of absolute
values of
weights
 Lasso coeff paths with decreasing λ

Source: Regression Shrinkage and Selection via the Lasso, by Robert Tibshirani, Journal of Royal Stat. Soc., 1996
 Compare to coeff shrinkage path of
ridge

Source: Sci-kit learn tutorial

 Contents
• Revisiting MSE and L2 regularization

• How L1 regularization lead to sparsity

• Other losses inspired by L1 and L2

• Hinge loss leads to small number of support vectors

• Link between logistic regression and cross entropy

 Smoothly Clipped Absolute Deviation
(SCAD) Penalty

Source: Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties, by Fan and Li, Journal of Am. Stat. Assoc., 2001
 Thresholding in three cases: No
alteration of large coefficients by SCAD
and Hard

Source: Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties, by Fan and Li, Journal of Am. Stat. Assoc., 2001
 Motivation for elastic net
• The p >> n problem and grouped selection
– Microarrays: p > 10,000 and n < 100.
– For those genes sharing the same biological “pathway”, the
correlations among them can be high.
• LASSO limitations
– If p > n, the lasso selects at most n variables. The number
of
– Grouped variables: the lasso fails to do grouped selection.
It tends to select one variable from a group and ignore the
others.

Source: Elastic net, by Zou and Hastie

 Elastic net: Use both L2 and L2
penalties

Source: Elastic net, by Zou and Hastie

 Geometry of elastic net

Source: Elastic net, by Zou and Hastie

 Elastic net selects correlated variables
as “group”

Source: Elastic net, by Zou and Hastie

 Elastic net selects correlated variables as
“group” and stabilizes the coefficient paths

Source: Elastic net, by Zou and Hastie

 Why L2 penalty keeps coefficients of
groups together?
• Try to think of an example with correlated
variables
 This analysis
can be
generalized
to linear
SVM

Source: Elastic SCAD SVM, by Becker, Toedt, Lichter and Benner, in BMC Bioinformatics2011
 A family of loss functions

Source: “A General and Adaptive Robust Loss Function” Jonathan T. Barron, ArXiv 2017
 A family of loss functions

Source: “A General and Adaptive Robust Loss Function” Jonathan T. Barron, ArXiv 2017
 Contents
• Revisiting MSE and L2 regularization

• How L1 regularization lead to sparsity

• Other losses inspired by L1 and L2

• Hinge loss leads to small number of support vectors

• Link between logistic regression and cross entropy

 What is hinge loss?
• Surrogate loss
function to 0-1 loss

• There are other

surrogate losses
possible
 Contents
• Revisiting MSE and L2 regularization

• How L1 regularization lead to sparsity

• Other losses inspired by L1 and L2

• Hinge loss leads to small number of support vectors

• Link between logistic regression and cross entropy

 Why logistic regression and BCE
• Let us assume a Bernoulli distribution
• 𝑃(𝑥) = 𝜇𝑥 (1 − 𝜇)1−𝑥
• An exponential family distribution is
𝜃𝑥−𝑏 𝜃
• 𝑃(𝑥|𝜃, 𝜑) = exp* + 𝑐 𝑥, 𝜑 +
𝑎 𝜑
• So, Bernoulli can be re-written as
μ
• 𝑃(𝑥) = exp 𝑥 log + log(1 − μ)
1−μ
μ
• Log odds of success is θ = log
1−μ
1
• So, μ =
1−𝑒 −θ
Source: Why the logistic function? A tutorial discussion on probabilities and neural networks, by Michael I. Jordan
 Generative vs. discriminative
Generative Discriminative
• Belief network A is more • More robust
modular
– Class-conditional densities are – Don’t need precise model
likely to be local, characteristic specification, so long as it is
functions of the ob jects being from exponential family
classiffied, invariant to the
nature and number of the • Requires fewer parameters
other classes
– O(n) as opposed to O(n2)
• More “natural”
– Deciding what kind of object to
generate and then generating it
from a recipe
• More efficient to estimate
mode, if correct

Source: Why the logistic function? A tutorial discussion on probabilities and neural networks, by Michael I. Jordan ftp://psyche.mit.edu/pub/jordan/uai.ps
 Losses for ranking and metric learning
• Margin loss

• Cosine similarity

• Ranking
– Point-wise
– Pair-wise
• φ(z) = (1-z)+, e-z, log(1-e-z)
– List-wise
Source: “Ranking Measures and Loss Functions in Learning to Rank” Chen et al, NIPS 2009
 Dropout: Drop a unit out to prevent
co-adaptation

Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, by Srivastava, Hinton, et al. in JMLR 2014.
 Why dropout?
• Make other features unreliable to break co-
adaptation
• Equivalent to adding noise
• Train several (dropped out) architectures in one
architecture (O(2n))
• Average architectures at run time
– Is this a good method for averaging?
– How about Bayesian averaging?
– Practically, this work well too

Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, by Srivastava, Hinton, et al. in JMLR 2014.
 Model averaging

• Average output should be the same

• Alternatively,
– w/p at training time
– w at testing time
Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, by Srivastava, Hinton, et al. in JMLR 2014.
 Difference between non-DO and DO
features

Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, by Srivastava, Hinton, et al. in JMLR 2014.
 Indeed, DO leads to sparse activation

Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, by Srivastava, Hinton, et al. in JMLR 2014.
 There is a sweet spot with DO, even if
you increase the number of neurons

Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, by Srivastava, Hinton, et al. in JMLR 2014.