Machine Learning: Principles and Techniques

Bagging, Boosting
IE 506

April 5, 2024

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 1 / 14

 Outline

1 Classification Algorithms
Bagging
Boosting

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 2 / 14

 Classification Algorithms Bagging

Classification Algorithms: Bagging

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 3 / 14

 Classification Algorithms Bagging

Bagging

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 4 / 14

 Classification Algorithms Bagging

Bagging

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 5 / 14

 Classification Algorithms Bagging

Bagging

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 6 / 14

 Classification Algorithms Boosting

Classification Algorithms: Boosting

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 7 / 14

 Classification Algorithms Boosting

Boosting: AdaBoost

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 8 / 14

 Classification Algorithms Boosting

AdaBoost - a loss perspective†

Input: N samples {(x i , y i )}N i d
i=1 , x ∈ R ,
i
y ∈ {+1, −1}, ∀i ∈ {1, 2, . . . , N}.

Initialize weights wi1 = 1/N, ∀i ∈ {1, 2, . . . , N}.

For t = 1, 2, . . . , T do:
▶ Train a weak classifier with examples weighed using current weights wit
PN
by minimizing: ϵt = i=1 wit I(ht (x i ) ̸= y i ).
▶ Compute αt = 1
2 ln 1−ϵ
ϵt
t

i i
▶ Update weights as: wit+1 = wit e −αt y h(x )
PN
▶ Normalize wit+1 = wit+1 / i=1 wit+1 .

Output: Final classifier h(x) = sign( T

P
t=1 αt ht (x)).

†
:J. Friedman, T. Hastie and R. Tibshirani. Additive logistic regression: A statistical
view of Boosting, Annals of Statistics, 2000, Vol. 28, no. 2, pp. 337–407.
P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 9 / 14
 Classification Algorithms Boosting

Bagging: AdaBoost

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 10 / 14

 Classification Algorithms Boosting

Bagging: AdaBoost

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 11 / 14

 Classification Algorithms Boosting

Bagging: AdaBoost

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 12 / 14

 Classification Algorithms Boosting

Bagging: AdaBoost

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 13 / 14

 Classification Algorithms Boosting

Bagging: AdaBoost

P. Balamurugan Machine Learning: Principles and Techniques April 5, 2024 14 / 14