1.

Ensemble Technique
1.1. General overview of ensemble technique
 Ensemble means ‘a group producing a single effect’
 In machine learning it is a technique that combines several
base models in orders to produce one optimal mode.
 In learning models noise variance and bias are the major
source of error.
 The ensemble methods in machine learning help minimize
these errors causing factors there by ensuring the accuracy
and stability of machine learning algorithms.
 CONT……
 Ensemble learning is a machine learning technique that
enhances accuracy and resilience in forecasting by merging
predictions from multiple models.
 The underlying concept behind ensemble learning is to
combine the outputs of diverse models to create a more precise
prediction.
 Ensemble techniques are a set of machine learning methods
that combine multiple models to improve the overall
performance of the learning system.
 How does Ensemble Learning Work?

 Assume you want to create a machine learning model that

predicts inventory stock orders for your company based on
historical data from previous years.
 You train four machine learning models using different
algorithms:
 Linear regression, support vector machine, regression decision
tree, and basic artificial neural network
 CONT……

 However, being weak does not imply being useless. You can
put them together to form an ensemble.
 For each new prediction, you run your input data through all
four models and then average the results.
 Because your machine learning models work differently,
ensemble learning is efficient. Each model may perform well
on some data but not on others.
 Why use Ensemble Learning?
• For several reasons, an ensemble is preferable to a single
model:
 Performance: As described in the preceding section, the
outcome of ensemble learning is a strong learner. The strong
learner is the result of weak learners. As a result, models'
predictive capabilities improve. Better performance is
achieved when compared to a single model.
 Error reduction: Machine learning model prediction errors
can be described by bias and variance. Bias is defined as the
difference between a prediction and the actual outcome.
 Variance is defined as a model's sensitivity to small changes in
the training set
 A model with low bias and variance is preferable
 Challenges of Ensemble Learning
 While ensemble learning is a powerful tool, and as fascinating
as it can look, it still does have some drawbacks.
 So below are some of the challenges faced by ensemble
learning.
 When you use an ensemble, you must devote more time and
resources to training your machine learning models.
 A random forest with 500 trees, for example, produces much
better results than a single decision tree, but it also takes much
longer to train
 CON,T
 Running ensemble models can also be
difficult if the algorithms you use require a
large amount of memory
 1.2. Describe types of ensemble technique
• There are two types of ensemble techniques:
1. Simple Ensemble Techniques
1. Max Voting
2. Averaging
3. Weighted Averaging
2. Advanced Ensemble techniques
1. Bagging
2. Boosting
3. Stacking
 1. Max Voting

 The max voting method is generally used for classification

problems.
 In this technique, multiple models are used to make
predictions for each data point.
 The predictions by each model are considered as a ‘vote’.
 The predictions which we get from the majority of the models
are used as the final prediction.
 CONT……….
 For example, when you asked 5 of your colleagues to rate your
movie (out of 5); we’ll assume three of them rated it as 4
while two of them gave it a 5.
 Since the majority gave a rating of 4, the final rating will be
taken as 4.
 You can consider this as taking the mode of all the predictions.
 The result of max voting would be something like this:
 2. Averaging

 Similar to the max voting technique, multiple predictions are

made for each data point in averaging.
 In this method, we take an average of predictions from all the
models and use it to make the final prediction.
 Averaging can be used for making predictions in regression
problems or while calculating probabilities for classification
problems.
 CONT………
 For example, in the below case, the averaging method would
take the average of all the values.
i.e. (5+4+5+4+4)/5 = 4.4
 3 Weighted Average
 This is an extension of the averaging method. All models are
assigned different weights defining the importance of each
model for prediction.
 For instance, if two of your colleagues are critics, while others
have no prior experience in this field, then the answers by these
two friends are given more importance as compared to the other
people.
 CONT……..

 The result is calculated as:-

[(5*0.23) + (4*0.23) + (5*0.18) + (4*0.18) + (4*0.18)] = 4.41

 2. Advanced Ensemble techniques
 Now that we have covered the basic ensemble techniques, let’s
move on to understanding the advanced techniques.

1. Bagging
 Bagging is a machine learning ensemble technique that can
improve the accuracy and stability of a model by generating
multiple subsets of the training data and training a separate
model on each subset using the same learning algorithm.
 CONT…..
 Examples of bagging algorithms include Bagged Decision
Trees.
 This method involves training multiple independent models on
different subsets of the training data and then aggregating their
predictions through averaging or voting.
 Bagging often reduces variance in the predictions and
improves the stability of the model.
 CONT…..
 The idea behind bagging is combining the results of multiple
models (for instance, all decision trees) to get a generalized
result.
 Here’s a question: If you create all the models on the same set of
data and combine it, will it be useful? There is a high chance that
these models will give the same result since they are getting the
same input.
 So how can we solve this problem? One of the techniques is
bootstrapping.
 Bootstrapping

 Bootstrapping is a sampling technique in which we create

subsets of observations from the original dataset, with
replacement.
 The size of the subsets is the same as the size of the original
set.
 Bagging (or Bootstrap Aggregating) technique uses these
subsets (bags) to get a fair idea of the distribution (complete
set).
 The size of subsets created for bagging may be less than the
original set
 Boosting
 Boosting is an ensemble technique that learns from previous
predictor errors in order to make better predictions in the
future.

 The technique combines several weak base learners to form

one strong learner, significantly improving model
predictability.

 Boosting works by arranging weak learners in a sequence so

that weak learners can learn from the next learner in the
sequence to create better predictive models.
 CONT…..
 New predictors are fitted to compensate for errors in previous
predictors.
 The gradient of descent aids the gradient booster in identifying
and correcting errors in learners' predictions.
 Boost improves speed and performance by utilizing decision
trees with boosted gradients.
 It heavily relies on the target model's computational speed and
performance.
 Model training should be done in a specific order, making
gradient boosted machine implementation time-consuming.
 Stacking
 Another ensemble method is stacking, which is also known as
stacked generalization.
 This technique works by allowing a training algorithm to
combine the predictions of several other similar learning
algorithms.
 Stacking has been used successfully in regression, density
estimation, distance learning, and classification.
 It can also be used to calculate the error rate during bagging.