Chapter 8-Part2

Boosting

1
- Simple (weak) classifiers are good

2
- Finding a classifier that’s
just right

3
Boosting question

4
Ensemble classifier

- A single classifier

5
- Ensemble method: each classifier “votes” on prediction

6
- Ensemble classifier in general

7
Boosting

Training a classifier

Learning decision stump

8
Boosting
Focus learning on
“hard” points

9
Learning on weighted data
More weight on “hard” or more important points

10
Learning a decision stump on weighted data

11
Boosting
Greedy learning ensembles
from data

12
AdaBoost algorithm

AdaBoost: learning ensembles

13
Computing coefficient wt of classifier ft(x)

14
15
Updating weights αi

16
Normalize weights αi

17
AdaBoost

18
AdaBoost example

t=1: just learning a classifier on original date

19
Updating weights

20
t=2: learn classifier on weighted data

21
Ensemble becomes weighted sum of learned classifiers

22
Decision boundary of ensemble classifier
after 30 iterations

23
Boosting convergence and overfitting

- After some iterations, training error of boosting goes to zero

24
AdaBoost Theorem

25
Condition of AdaBoost Theorem

26
27
Boosting tends to be robust to overfitting

28
But boosting will eventually overfit, so must choose max number of
components T

29
30
Threshold split selection algorithm

31
Visualizing the threshold split

32
Split on Age >=38

33
Depth 2: split on Income >=$60k

34
Each split partitions the 2-D space

35
Decision trees vs logistic regression

Logistic regression

36
Decision tree
- Deep 1: Split on
x[1]

37
- Deep 2: split on x[2]

38
39
Decision boundaries

40
Comparing decision boundaries

41