Table of Contents

Topic1: Introduction to Data Mining, task types, input ............................................................ 1

Topic 2: 1R, ID3, C4.5 tree learning algorithm ......................................................................... 9
Topic 3: Association Rule Mining, apriori algorithm .............................................................. 16
Topic 4: Linear regression, "regression" (predicting numeric values) ................................... 17
Topic 5: Support Vector Machines ......................................................................................... 22
Topic 6: Bayesian networks, Naïve Bayes.............................................................................. 27
Topic 7: Instance based learning ........................................................................................... 34
Topic 8: Clustering ................................................................................................................ 35
Topic 9: Data transformation, hyperparameter tuning .......................................................... 39
Topic 10: Ensemble learning ................................................................................................. 44
Topic 11: Logistic regression, Perceptron, Winnow, Neural networks ..................................... 49
Topic 12: Performance evaluation – measures ...................................................................... 56
Topic 13: Performance evaluation – datasets ........................................................................ 59

Topic1: Introduction to Data Mining, task types, input

Input
In DM & ML, the input to the learning scheme is a set of instances or a dataset, which is
represented as a single flat table with one line (vector) for each instance. Instances are specific
types of examples that are independent and characterized by a predetermined set of attributes.
The types of attributes include nominal, ordinal, interval, and ratio, and they impact the
learning scheme differently. Data cleansing and integration are critical for assembling,
integrating, and cleaning up data from different sources. Simple visualization tools are useful for
identifying problems, and domain experts should be consulted for data inspection.

Attribute types
1. Nominal Attributes:
- Nominal attributes represent categories without any inherent order or ranking.
- Examples include colors, shapes, or any other categorical data without a specific order.
- In DM & ML, nominal attributes are often used for classification tasks.

2. Ordinal Attributes:
 - Ordinal attributes represent categories with a clear order or ranking.
- Examples include rankings (e.g., low, medium, high) or survey responses (e.g., strongly
disagree, disagree, neutral, agree, strongly agree).
- In DM & ML, ordinal attributes can be used for tasks where the order of categories matters,
such as ranking or prioritization.

3. Interval Attributes:
- Interval attributes represent numerical values where the difference between values is
meaningful, but there is no true zero point.
- Examples include temperature in Celsius or Fahrenheit.
- In DM & ML, interval attributes are used for tasks where the difference between values is
important, but the absence of a true zero point means that ratios are not meaningful.

4. Ratio Attributes:
- Ratio attributes represent numerical values where both the difference between values and
the ratio of values are meaningful, and there is a true zero point.
- Examples include weight, height, and income.
- In DM & ML, ratio attributes are used for tasks where both differences and ratios between
values are meaningful, such as in regression analysis or certain types of clustering.

Understanding the different types of attribute measurements is crucial for data preprocessing,
feature engineering, and selecting appropriate algorithms for DM & ML tasks.

Attribute type conversions

Attribute type conversions are a common data transformation technique in DM & ML that
involve converting attributes from one type to another. This is often necessary because different
learning algorithms may require different types of attributes. Here are some examples of
attribute type conversions:

1. Nominal to Numeric:
- This conversion involves assigning integers to nominal values.
- For example, if a nominal attribute has values "red", "green", and "blue", these could be
converted to 1, 2, and 3, respectively.
- However, it is important to note that this conversion can result in false assumptions about
order or difference on numeric attributes.

2. Ordinal to Nominal:
- This conversion involves converting ordinal attributes to nominal attributes.
- For example, if an ordinal attribute has values "low", "medium", and "high", these could be
converted to nominal values "1", "2", and "3", respectively.
- However, it is important to note that information on the order of values is lost in this
conversion.

3. Nominal to Binary-Nominal:
 - This conversion involves creating separate binary attributes for each nominal value.
- For example, if a nominal attribute has values "red", "green", and "blue", three binary
attributes could be created: "is_red", "is_green", and "is_blue".
- This conversion can be useful for certain types of classification tasks.

4. Ordinal to n-1 Boolean Attributes:

- This conversion involves creating n-1 Boolean attributes for an ordinal attribute with n
values.
- For example, if an ordinal attribute has values "low", "medium", and "high", two Boolean
attributes could be created: "is_low_or_medium" and "is_high".
- This conversion can be useful for certain types of classification tasks.

Attribute type conversions can be performed using various techniques and libraries in DM & ML,
such as the Python scikit-learn library's OrdinalEncoder and OneHotEncoder classes. However, it
is important to carefully consider the implications of attribute type conversions and choose the
appropriate conversion technique for the specific dataset and task at hand.

Inaccurate values

Inaccurate values in a dataset can cause problems in DM & ML because they can lead to
incorrect or biased results. Here are some specific problems that can arise from inaccurate
values:

1. Errors and Omissions:

- Inaccurate values can result from errors or omissions in data collection.
- For example, if a survey respondent accidentally selects the wrong response, this can result
in inaccurate data.
- These errors and omissions can lead to incorrect conclusions or predictions.

2. Typographical Errors:
- Inaccurate values can also result from typographical errors in nominal attributes.
- For example, if a product name is misspelled, this can result in inconsistent data.
- These errors can lead to problems with data integration and analysis.

3. Measurement Errors:
- Inaccurate values can also result from measurement errors in numeric attributes.
- For example, if a weight measurement is incorrect, this can result in outliers or biased data.
- These errors can lead to problems with data analysis and modeling.

4. Duplicates and Stale Data:

- Inaccurate values can also result from duplicates or stale data in a dataset.
- For example, if a customer record is duplicated, this can result in inaccurate counts or biased
data.
- These errors can lead to problems with data analysis and modeling.
To address these problems, it is important to perform data cleansing and integration to identify
and correct inaccurate values in a dataset. This can involve techniques such as outlier detection,
duplicate removal, and data validation. By addressing inaccurate values, DM & ML practitioners
can ensure that their results are accurate and unbiased.

Missing values

Missing values in a dataset can pose significant challenges in DM & ML, as they can lead to
biased or inaccurate results. Here are some specific issues associated with missing values:

1. Biased Analysis:
- Missing values can lead to biased analysis if not handled properly. For example, if certain
demographic information is missing from a customer dataset, it can lead to biased conclusions
about customer behavior or preferences.

2. Reduced Sample Size:

- Missing values can reduce the effective sample size for analysis, which can impact the
statistical power of models and lead to less reliable results.

3. Inaccurate Imputation:
- Imputing missing values without careful consideration can introduce inaccuracies into the
dataset. For example, using mean imputation for a variable with a significant number of missing
values can distort the distribution of the data.

4. Misinterpretation of Missingness:
- The reason for missing values can vary, and misinterpreting the nature of missingness can
lead to incorrect conclusions. For instance, assuming that missing values are completely at
random when they are actually related to certain characteristics can bias the analysis.

To address the challenges associated with missing values, various techniques can be employed,
including:

- Imputation: Imputing missing values using statistical methods such as mean, median, mode
imputation, or more advanced techniques like k-nearest neighbors (KNN) imputation or multiple
imputation.

- Deletion: Removing records or variables with missing values, though this approach should be
used judiciously to avoid significant loss of information.

- Advanced Modeling: Using models that can handle missing data directly, such as decision trees
or random forests, or employing techniques like maximum likelihood estimation in statistical
modeling.
By carefully addressing missing values through appropriate techniques, DM & ML practitioners
can mitigate the impact of missing data on their analyses and models, leading to more accurate
and reliable results.

Task types
In machine learning, there are three main types of tasks that can be performed:

1. **Supervised Learning:** In supervised learning, the algorithm is trained on a labeled

dataset, where the input data is associated with corresponding output labels. The goal is to
learn a mapping between the input and output variables, so that the algorithm can predict the
output for new, unseen input data. Examples of supervised learning tasks include classification,
regression, and sequence prediction.

2. **Unsupervised Learning:** In unsupervised learning, the algorithm is trained on an

unlabeled dataset, where the input data is not associated with any output labels. The goal is to
discover patterns, structures, and relationships in the data, such as clusters, associations, and
anomalies. Examples of unsupervised learning tasks include clustering, dimensionality
reduction, and anomaly detection.

3. **Reinforcement Learning:** In reinforcement learning, the algorithm learns to make

decisions based on feedback from the environment. The goal is to learn a policy that maximizes
a reward signal, which is provided by the environment in response to the agent's actions.
Examples of reinforcement learning tasks include game playing, robotics, and autonomous
driving.

Each of these task types has its own set of algorithms, techniques, and evaluation metrics, and
the choice of task type depends on the nature of the problem and the available data.

Data transformation

In DM & ML, data transformation is a crucial step in data preprocessing, which involves
converting raw data into a format that can be used for analysis. Here are some common data
transformations that can be performed and how they can be done:

1. Normalization:
- Normalization is the process of scaling numeric data to a common range, typically between 0
and 1.
- This can be done using various techniques, such as min-max scaling or z-score normalization.

2. Binning:
- Binning is the process of grouping numeric data into discrete bins or categories.
- This can be done using various techniques, such as equal width or equal frequency binning.
3. One-Hot Encoding:
- One-hot encoding is the process of converting categorical data into binary vectors.
- This can be done using various techniques, such as pandas.get_dummies() in Python or
OneHotEncoder in scikit-learn.

4. Feature Scaling:
- Feature scaling is the process of scaling numeric data to a common range, typically between -
1 and 1.
- This can be done using various techniques, such as standardization or normalization.

5. Feature Selection:
- Feature selection is the process of selecting a subset of relevant features from a larger set of
features.
- This can be done using various techniques, such as correlation analysis or feature importance
ranking.

6. Dimensionality Reduction:
- Dimensionality reduction is the process of reducing the number of features in a dataset while
preserving as much information as possible.
- This can be done using various techniques, such as principal component analysis (PCA) or t-
distributed stochastic neighbor embedding (t-SNE).

These are just a few examples of the many data transformations that can be performed in DM &
ML. The choice of transformation depends on the specific dataset and the task at hand.

Denormalization
Denormalization is a process of transforming a normalized database schema into a less
normalized schema, typically for performance reasons. In a normalized schema, data is
organized into multiple tables to minimize redundancy and improve data consistency. However,
this can result in complex queries and slower performance when dealing with large datasets.
Denormalization involves combining tables and duplicating data to simplify queries and improve
performance.

For example, consider a database schema with separate tables for customers, orders, and order
items. To retrieve all orders for a particular customer, a query would need to join the customer,
order, and order item tables. This can be time-consuming for large datasets. By denormalizing
the schema and adding customer information to the order table, the query can be simplified
and executed more quickly.

While denormalization can improve performance, it can also introduce data redundancy and
inconsistency. Therefore, it should be used judiciously and with careful consideration of the
trade-offs between performance and data integrity.

Data Cleansing
Data cleansing, also known as data cleaning, is the process of identifying and correcting errors,
inconsistencies, and inaccuracies in a dataset to improve its quality and reliability for analysis
and modeling in data mining and machine learning.

The data cleansing process typically involves several key steps:

1. **Identifying Errors:** This involves detecting and identifying various types of errors in the
dataset, such as missing values, outliers, duplicate records, and inconsistencies in data formats.

2. **Handling Missing Values:** Addressing missing values by imputing them using statistical
methods, removing records with missing values, or employing advanced imputation techniques
to fill in the gaps.

3. **Dealing with Outliers:** Identifying and handling outliers that may skew the analysis or
modeling results. This can involve removing outliers or transforming them to reduce their
impact.

4. **Resolving Duplicate Records:** Identifying and removing duplicate records to ensure that
each observation in the dataset is unique.

5. **Standardizing Data Formats:** Ensuring consistency in data formats, such as date formats,
numerical representations, and categorical values, to facilitate accurate analysis and modeling.

6. **Correcting Inconsistencies:** Addressing inconsistencies in data, such as conflicting

information across different attributes or records, to improve data integrity.

7. **Validating Data:** Verifying the accuracy and validity of the data by comparing it against
known standards or business rules.

By performing data cleansing, practitioners can enhance the quality of the dataset, reduce the
risk of biased or inaccurate analysis, and improve the reliability of the results obtained from
data mining and machine learning processes.

Data integration
Data integration is the process of combining data from different sources into a unified view,
providing a comprehensive and consistent representation of the data for analysis, reporting,
and decision-making purposes in data mining and machine learning.

Key aspects of data integration include:

1. **Assembling Data:** Gathering data from diverse sources such as databases, data
warehouses, spreadsheets, and external systems.
2. **Consolidating Data:** Bringing together data from disparate sources and unifying it into a
coherent and consistent format, often involving the transformation of data to ensure
compatibility and standardization.

3. **Cleaning Data:** Addressing inconsistencies, errors, and redundancies in the data to

improve its quality and reliability, often through data cleansing techniques.

4. **Resolving Data Conflicts:** Identifying and resolving conflicts that arise when integrating
data from different sources, such as conflicting attribute names or data formats.

5. **Establishing Data Relationships:** Establishing relationships and associations between data

elements from different sources to create a unified view of the data.

6. **Creating a Single Source of Truth:** Developing a single, authoritative source of integrated

data, often in the form of a data warehouse or data mart, to serve as a consistent point of
access for analysis and reporting.

Data integration is critical for enabling organizations to derive meaningful insights from their
data by providing a unified and comprehensive view of information that was previously
scattered across multiple systems. It plays a fundamental role in supporting data-driven
decision-making and facilitating the use of data mining and machine learning techniques to
extract valuable knowledge and patterns from integrated datasets.
Overfitting occurs when a model learns the training data too well, capturing noise and random
fluctuations, which reduces its ability to generalize to new, unseen data. This can lead to poor
performance on test data.

Underfitting, on the other hand, happens when a model is too simple to capture the underlying
patterns in the data, resulting in poor performance on both the training and test data.

Bias refers to the error introduced by approximating a real-world problem, which can lead to
the model missing relevant relations between features and target outputs. High bias can result
in underfitting.

Variance measures the model's sensitivity to fluctuations in the training data. High variance can
lead to overfitting, as the model is capturing noise and not the underlying patterns in the data.

In summary:
- Overfitting: Model learns noise in the training data, leading to poor generalization.
- Underfitting: Model is too simple to capture the underlying patterns in the data, resulting in
poor performance.
- Bias: Error introduced by approximating a real-world problem, leading to missing relevant
relations.
- Variance: Model's sensitivity to fluctuations in the training data, which can lead to overfitting.
Balancing bias and variance is crucial for building models that generalize well to new data.

Topic 2: 1R, ID3, C4.5 tree learning algorithm

White box and black box algorithms refer to different levels of transparency and
visibility into the internal workings of a computational model:

1. **White Box Algorithm:**

- Also known as transparent or glass-box algorithms.
- The internal logic and mechanisms of the algorithm are fully understandable and
visible.
- Users have access to information about how the algorithm makes decisions.
- Examples include linear regression and decision trees.

2. **Black Box Algorithm:**

- Also known as opaque
- The internal workings of the algorithm are not transparent or easily interpretable.
- Users may not have insight into how the algorithm arrives at specific decisions.
- Examples include complex neural networks and some machine learning models
where the focus is on performance rather than interpretability.

The choice between white box and black box algorithms often depends on the specific
requirements of a task. White box models are preferred when interpretability and
understanding the decision-making process are crucial, such as in certain regulatory or
sensitive applications. Black box models, on the other hand, may be chosen for tasks
where achieving high accuracy is the primary goal, and understanding the underlying
process is less important.

1R
The 1R algorithm, also known as "One Rule," is a simple and intuitive classification
algorithm. It works by selecting a single attribute (feature) and creating a single rule
based on that attribute to make predictions. Here's a brief overview of how the 1R
algorithm works:

1. For each attribute:

- For each value of the attribute, the algorithm counts how often each class appears.
- It finds the most frequent class for each attribute value.

2. The algorithm then creates a rule for the attribute based on the most frequent class:
- The rule assigns the most frequent class to the attribute value.
3. After creating rules for all attributes, the algorithm calculates the error rate of the
rules.

4. The algorithm chooses the rules with the smallest error rate.

In essence, the 1R algorithm selects the attribute that minimizes the error rate when
used to predict the class, making it a straightforward and easy-to-understand approach
to classification.

The 1R algorithm is primarily designed for nominal attributes, but it can be adapted to
handle numeric values through a process called discretization. Discretization involves
dividing the range of numeric values into intervals or ranges and then treating these
intervals as nominal values. Here's how you can handle numeric values with the 1R
algorithm:

1. Discretization:
- Divide the numeric attribute into a set of intervals or ranges. This can be done using
various techniques such as equal width binning, equal frequency binning, or more
advanced methods like entropy-based binning.

2. Convert numeric values to nominal values:

- Map each numeric value to the interval it falls into. This effectively converts the
numeric attribute into a set of nominal values.

3. Apply the 1R algorithm:

- Once the numeric attribute has been discretized into nominal values, you can apply
the standard 1R algorithm as described earlier.

By discretizing numeric attributes, you can use the 1R algorithm to handle numeric
values and incorporate them into the classification process. Keep in mind that the
choice of discretization method can impact the performance of the algorithm, and it's
important to consider the characteristics of the data when performing discretization.

Discretization procedure is very sensitive to noise

– A single instance with an incorrect class label will
most likely result in a separate interval
• Simple solution: enforce minimum number of
instances in majority class per interval
• Weather data example (with minimum set to 3):
////
64 65 68 69 70 71 72 72 75 75 80 81 83 85
Yes |No |Yes Yes Yes |No No |Yes Yes Yes |No |Yes Yes |No
64 65 68 69 70 71 72 72 75 75 80 81 83 85
Yes No Yes Yes Yes |No No Yes Yes Yes |No Yes Yes No
/

The 1R algorithm is a simple and intuitive classification algorithm that is easy to

understand and implement. Here are some advantages and disadvantages of the 1R
algorithm:

Advantages:
- Simplicity: The 1R algorithm is easy to understand and implement, making it a good
choice for beginners or for situations where a quick and simple solution is needed.
- Speed: The algorithm is fast and efficient, making it suitable for large datasets.
- Interpretable: The rules generated by the algorithm are easy to interpret and can
provide insights into the data.

Disadvantages:
- Limited applicability: The 1R algorithm is primarily designed for nominal attributes and
may not work well with continuous or numeric data.
- Overfitting: The algorithm tends to overfit the training data, which can lead to poor
generalization performance on new data.
- Sensitivity to attribute selection: The performance of the algorithm is highly dependent
on the choice of attribute used to create the rules. If the wrong attribute is selected, the
algorithm may perform poorly.

Overall, the 1R algorithm is a simple and effective classification algorithm that can be
useful in certain situations. However, it has some limitations and may not be suitable for
all types of data or classification problems. It is important to carefully consider the
characteristics of the data and the goals of the analysis before deciding to use the 1R.

ID3, greedy algorithm, white box

The ID3 (Iterative Dichotomiser 3) algorithm is a popular decision tree algorithm used
for classification. Decision trees are a type of supervised learning algorithm that is used
for both classification and regression tasks. The ID3 algorithm builds a decision tree by
recursively partitioning the dataset based on the attributes that best separate the
classes or minimize impurity.

Here's an overview of how the ID3 algorithm works:

1. Select the best attribute: The algorithm selects the attribute that best separates the
classes or minimizes impurity. This is typically done using measures such as information
gain or Gini impurity.

2. Partition the dataset: The dataset is partitioned into subsets based on the values of
the selected attribute.

3. Recur: The algorithm recursively applies the same process to each subset, creating
branches in the decision tree.

4. Stop criteria: The recursion stops when one of the following conditions is met:
- All instances in a subset belong to the same class.
- There are no more attributes to use for further splitting.

5. Create leaf nodes: Once the recursion stops, the algorithm creates leaf nodes that
represent the class labels.

The resulting decision tree can be used to make predictions by following the branches
based on the attribute values of a given instance until a leaf node is reached, which
provides the predicted class label.

Advantages of decision trees and the ID3 algorithm:

- Interpretability: Decision trees are easy to interpret and visualize, making them useful
for understanding the underlying decision-making process.
- Handling non-linear relationships: Decision trees can capture non-linear relationships
between features and the target variable.
- Feature selection: The algorithm implicitly performs feature selection by choosing the
most informative attributes for splitting.

Disadvantages:
- Overfitting: Decision trees are prone to overfitting, especially when the tree grows too
deep or when the dataset is noisy.
- Instability: Small variations in the data can lead to different decision trees, making
them less stable compared to other algorithms.
The ID3 algorithm and decision trees in general are powerful tools for classification
tasks, and they have been extended and improved upon by other algorithms such as
C4.5, CART, and random forests.

Popular impurity criterion: information gain

– Information gain increases with the average
„purity” of the subsets that an attribute produces
• Strategy: choose attribute that results in greatest
information gain
Information gain:
information before splitting – information after splitting
gain(Outlook ) = info([9,5]) – info([2,3],[4,0],[3,2])
= 0.940 – 0.693 = 0.247 bits
Entropy is a good measure for purity because it captures the degree of uncertainty or
randomness in a set of data. In the context of decision trees and classification, entropy
is used to measure the impurity of a set of instances with respect to their class labels.
The higher the entropy, the more mixed the set of instances is with respect to their class
labels, and the lower the entropy, the more pure the set of instances is with respect to
their class labels.

Here are some reasons why entropy is a good measure for purity:

1. Intuitive interpretation: Entropy has an intuitive interpretation as a measure of

uncertainty or randomness. It captures the degree to which the class labels are mixed or
uncertain in a set of instances.

2. Satisfies desirable properties: Entropy satisfies desirable properties for a measure of

purity, such as being maximal when all classes are equally likely and being minimal when
all instances belong to the same class.

3. Considers all classes: Entropy considers all classes in the dataset, not just the majority
class. This is important because it allows the algorithm to identify attributes that are
informative for separating minority classes.

4. Compatible with information gain: Entropy is used in conjunction with information

gain to select the best attribute for splitting the data. Information gain measures the
reduction in entropy achieved by splitting the data on a particular attribute, and it is
used to select the attribute that maximizes the reduction in entropy.
Overall, entropy is a good measure for purity because it captures the degree of
uncertainty or randomness in a set of data and satisfies desirable properties for a
measure of purity. It is widely used in decision tree algorithms such as ID3 and C4.5 to
select the best attribute for splitting the data.

Information is measured in bits

– Given a probability distribution, the info required to
predict an event is the distribution’s entropy
– Entropy gives the information required in bits (this can
involve fractions of bits!)

Subsets are more likely to be pure if there

is a large number of values
Information gain is biased towards choosing
attributes with a large number of values
This may result in overfitting (selection of an
attribute that is non-optimal for prediction)

Solution: Gain ratio: a modification of the information gain that

reduces its bias towards highly branching attributes
• Gain ratio takes the number and the size of
branches into account when choosing an attribute
(disregarding the class values!)
– It corrects the information gain by taking the
intrinsic information of a split into account
• Intrinsic information: entropy of distributing instances
into branches (i.e., how much info do we need to tell
which branch an instance belongs to)

C4.5
According to ,[ ],, the ID3 algorithm is a top-down induction of decision trees that was
developed by Ross Quinlan. It is a simple algorithm that works by recursively partitioning the
data based on the attribute that provides the most information gain. However, ID3 has some
limitations, such as its inability to handle numeric attributes, missing values, and noisy data.

To address these limitations, Quinlan developed the C4.5 algorithm, which is an extension of
ID3. C4.5 can handle numeric attributes, missing values, and noisy data by using statistical
methods to estimate the probabilities of different outcomes. Additionally, C4.5 uses a more
sophisticated attribute selection criterion called gain ratio, which takes into account the number
of distinct values an attribute can take on. Overall, C4.5 is a more robust and flexible algorithm
than ID3.
C4.5 is a decision tree algorithm that works by recursively partitioning the data based on the
attribute that provides the most gain ratio. Gain ratio is a modification of information gain that
takes into account the number of distinct values an attribute can take on. C4.5 can handle both
discrete and continuous attributes, missing values, and noisy data. It also has the ability to
prune the decision tree to avoid overfitting.

The advantages of C4.5 include its ability to handle a wide range of data types and its ability to
prune the decision tree to avoid overfitting. Additionally, C4.5 produces decision trees that are
easy to interpret and can be used to generate classification rules.

The disadvantages of C4.5 include its tendency to create biased trees when the data is
imbalanced, and its sensitivity to irrelevant attributes. Additionally, C4.5 can be computationally
expensive when dealing with large datasets or complex decision trees.

Pruning is a technique used in decision tree learning to prevent overfitting and improve the
generalization performance of the model. Overfitting occurs when a decision tree captures
noise in the training data and makes the model less effective at making predictions on new,
unseen data. Pruning helps to simplify the decision tree by removing parts of the tree that do
not contribute significantly to its predictive accuracy.

There are two main types of pruning in decision tree learning:

1. **Prepruning**: Prepruning involves stopping the growth of the tree early, before it becomes
overly complex. This can be achieved by setting a limit on the maximum depth of the tree, the
minimum number of samples required to split a node, or the maximum number of leaf nodes.
Prepruning is based on stopping criteria such as statistical significance tests, which halt the
growth of the tree when further expansion does not lead to a statistically significant
improvement in predictive accuracy ,[ ],.

2. **Postpruning**: Postpruning, also known as backward pruning, involves growing the full
decision tree and then removing or collapsing nodes that do not provide significant predictive
power. This is typically done by replacing or raising subtrees within the larger tree, based on a
pruning strategy such as error estimation ,[ ],. Postpruning aims to simplify the tree by
removing branches that are not informative or may be capturing noise in the training data.

Pruning works by reducing the complexity of the decision tree, which in turn reduces the risk of
overfitting. By simplifying the tree, pruning helps to improve the model's ability to generalize to
new, unseen data. It also makes the decision tree more interpretable and easier to understand.

In summary, pruning is a technique used in decision tree learning to prevent overfitting by

simplifying the tree structure. It can be achieved through prepruning, which stops the growth of
the tree early, or postpruning, which simplifies a fully grown tree by removing less informative
branches ,[ ],.
Topic 3: Association Rule Mining, apriori algorithm
Association rule mining is a data mining technique used to discover interesting relationships or
associations among a set of items in large datasets. The most common application of
association rule mining is in market basket analysis, where it is used to identify patterns of co-
occurrence of items in transactions, such as "if a customer buys product A, they are likely to buy
product B as well."

The algorithm for association rule mining typically involves two key metrics: support and
confidence. Support measures the frequency of co-occurrence of items in the dataset, while
confidence measures the reliability of the association rule.

The process of association rule mining involves identifying frequent itemsets (sets of items that
frequently occur together) and then generating association rules from these itemsets. These
rules are then evaluated based on their support and confidence to identify meaningful and
actionable associations.
Supp(X and Y) „Itemset” – no matter, whether an item (attribute)
is on the left or on the right-hand side of the rule
confidence=Supp(X and Y)/Supp(X)
Lift: How popular is Y, when X, considering also how popular Y
is
– lift= Confidence of Rule / Relative support of Y
– 1: no association,
>1 Y is more likely when X,
<1 Y is less likely when X

The uses of association rule mining include:

1. Market basket analysis in retail to understand customer purchasing behavior and optimize
product placement and promotions.
2. Cross-selling and upselling in e-commerce to recommend related products to customers
based on their purchase history.
3. Healthcare analytics to identify patterns in patient diagnoses and treatments.

However, association rule mining also has limitations:

1. It may generate a large number of rules, making it challenging to identify the most relevant
and actionable associations.
2. The algorithm may not perform well with sparse datasets or datasets with a large number of
items.
3. It may not capture complex relationships or dependencies among items in the dataset.

Overall, association rule mining is a powerful technique for discovering interesting patterns and
relationships in large datasets, but it requires careful interpretation and evaluation of the
generated rules to derive meaningful insights.
The Apriori algorithm is a classic algorithm used for association rule mining in large datasets. It
is based on the idea that if an itemset is frequent, then all of its subsets must also be frequent.
The algorithm works by iteratively generating candidate itemsets of increasing size and pruning
those that do not meet the minimum support threshold.

The Apriori algorithm consists of two main steps: the generation of frequent itemsets and the
generation of association rules.

1. Generation of frequent itemsets:

The algorithm starts by scanning the dataset to identify the support of each item. The support
of an item is the number of transactions in which it appears. The algorithm then generates
frequent itemsets of size k by joining frequent itemsets of size k-1. The join operation involves
taking two frequent itemsets of size k-1 and combining them to form a new itemset of size k.
The resulting itemset is then pruned if any of its subsets are not frequent.

2. Generation of association rules:

Once the frequent itemsets have been generated, the algorithm generates association rules by
considering all possible subsets of each frequent itemset. For each frequent itemset, the
algorithm generates all possible non-empty subsets and computes the confidence of each rule.
The confidence of a rule is the ratio of the support of the itemset containing both the
antecedent and consequent to the support of the antecedent. The algorithm then prunes the
rules that do not meet the minimum confidence threshold.

The Apriori algorithm has several advantages, including its simplicity and efficiency in handling
large datasets. However, it also has some limitations, such as its inability to handle datasets with
a large number of items or a low minimum support threshold. Additionally, the algorithm may
generate a large number of candidate itemsets, making it challenging to identify the most
relevant and actionable associations.

Brute force: All rules, filtering on the bases of support and

confidence
• Immense number
of possible associations
• Computational complexity!

Topic 4: Linear regression, "regression" (predicting numeric values)

Inputs (attribute values) and output are all numeric
• Output is the sum of weighted attribute values (linear
combination)
–
The trick is to find good values for the weights
Linear regression makes several key assumptions:
1. Linearity: The relationship between the independent variables and the dependent variable is
linear.
2. Independence: The residuals (the differences between the observed and predicted values) are
independent of each other.
3. Homoscedasticity: The variance of the residuals is constant across all levels of the
independent variables.
4. Normality: The residuals are normally distributed.

The model is trained by finding the coefficients for the independent variables that minimize the
difference between the predicted and actual values of the dependent variable. This is typically
done using the method of least squares, which aims to minimize the sum of the squared
differences between the observed and predicted values.

Advantages of linear regression include its simplicity, interpretability, and the ability to provide
insights into the relationships between variables. It is also computationally efficient and can be
applied to both continuous and categorical independent variables.

Disadvantages of linear regression include its assumption of linearity, which may not hold in all
cases, and its sensitivity to outliers. Additionally, it may not capture complex, non-linear
relationships between variables.

Mean Squared Error (MSE) and Absolute Error (also known as Mean Absolute Error, MAE) are
both metrics used to evaluate the performance of regression models. The main difference
between the two is in how they measure the errors. MSE squares the errors before averaging
them, which gives more weight to large errors and makes the metric sensitive to outliers. On the
other hand, MAE takes the absolute value of the errors before averaging them, which makes it
less sensitive to outliers.
• Minimization of absolute error is more difficult!

Linear models can be used for binary classification through techniques such as logistic
regression. In logistic regression, the model predicts the probability of an observation belonging
to a certain class using a logistic function. The model is trained to maximize the likelihood of the
observed data given the model parameters, and it outputs the probability of the observation
belonging to the positive class. The decision boundary is typically set at a probability threshold
of 0.5, above which the observation is classified as belonging to the positive class, and below
which it is classified as belonging to the negative class.
Binary classification
• Prediction is made by plugging in observed
values of the attributes into the expression
– Predict one class if output > 0 (or 0.5, ...), and
the other class if output < 0 (or 0.5, ...)
• Decision boundary
– defines where the decision changes from one class
value to the other
– hyperplane (high-dimensional plane)
Regularized linear regression, also known as ridge regression, is a technique used to
mitigate the problem of overfitting in linear regression models. In traditional linear
regression, the model aims to minimize the sum of squared differences between the
observed and predicted values. However, when the number of features is large,
traditional linear regression can lead to overfitting, where the model fits the noise in the
data rather than the underlying pattern.

Ridge regression addresses this issue by adding a penalty term to the traditional linear
regression ive function. The ive function of ridge regression is to minimize the residual
sum of squares (RSS) along with a penalty term, which is the L2 norm (sum of the
squared coefficients) multiplied by a regularization parameter (alpha).

The addition of the penalty term encourages the model to not only minimize the errors
between the predicted and actual values but also to keep the coefficients of the
features small. This helps in reducing the model's complexity and mitigates the impact
of multicollinearity among the predictor variables.

Ridge regression is particularly useful when dealing with high-dimensional datasets or

when multicollinearity is present among the predictor variables. By controlling the
complexity of the model, ridge regression can improve its generalization performance
and reduce the risk of overfitting.

In summary, ridge regression, is a technique that adds a penalty term to the traditional
linear regression ive function to address overfitting and multicollinearity, leading to
more robust and generalizable models

Lasso regression, or Least Absolute Shrinkage and Selection Operator, is another

technique used to mitigate the problem of overfitting in linear regression models.
Similar to ridge regression, lasso regression adds a penalty term to the traditional linear
regression ive function. However, instead of using the L2 norm of the coefficients, lasso
regression uses the L1 norm (sum of the absolute values of the coefficients) multiplied
by a regularization parameter (alpha).

The L1 penalty term encourages the model to not only minimize the errors between the
predicted and actual values but also to set some of the coefficients to zero. This leads to
feature selection, where some of the less important features are eliminated from the
model, resulting in a simpler and more interpretable model.
The main difference between ridge and lasso regression is the type of penalty term
used. Ridge regression uses the L2 norm penalty, which shrinks the coefficients towards
zero, but does not set any of them exactly to zero. On the other hand, lasso regression
uses the L1 norm penalty, which can set some of the coefficients exactly to zero, leading
to feature selection.

Another difference between ridge and lasso regression is their behavior when dealing
with highly correlated features. Ridge regression can handle correlated features by
shrinking their coefficients towards each other, while lasso regression may arbitrarily
select one of the correlated features and eliminate the others.

In summary, lasso regression is a technique used to address overfitting in linear

regression models by adding an L1 norm penalty term to the ive function. It leads to
feature selection and a simpler model compared to ridge regression, which uses an L2
norm penalty term. However, lasso regression may arbitrarily select one of the
correlated features and eliminate the others, while ridge regression can handle
correlated features by shrinking their coefficients towards each other

Regularization is a technique used in machine learning to prevent overfitting and

improve the generalization of models. It involves adding a penalty term to the model's
ive function, which encourages the model to avoid learning complex patterns from the
training data that may not generalize well to new, unseen data.

L1 and L2 regularization are two common types of regularization techniques used in

linear regression and other models:

1. L1 Regularization (Lasso):
- L1 regularization adds a penalty term to the model's ive function equal to the sum of
the absolute values of the coefficients multiplied by a regularization parameter (alpha).
- It encourages sparsity in the model by driving some of the coefficients to exactly
zero, effectively performing feature selection.
- L1 regularization is particularly useful when the dataset contains many irrelevant or
redundant features.

2. L2 Regularization (Ridge):
- L2 regularization adds a penalty term to the model's ive function equal to the sum of
the squared values of the coefficients multiplied by a regularization parameter (alpha).
- It discourages large coefficients and effectively shrinks them towards zero, but
generally does not force them to be exactly zero.
 - L2 regularization is effective at handling multicollinearity and reducing the impact of
irrelevant features on the model.

In summary, regularization is a technique used to prevent overfitting by adding a

penalty to the model's ive function, and it comes in different forms such as L1 and L2
regularization. L1 regularization (Lasso) encourages sparsity and feature selection, while
L2 regularization (Ridge) discourages large coefficients and is effective at handling
multicollinearity.

Counterparts exist for (almost all) classification

algorithms
• All classification algorithms can be applied to
regression problems using discretization
(almost)
– Prediction: weighted average of intervals’
midpoints (weighted according to class
probabilities)
• Regression more difficult than classification (i.e.,
percent correct vs. mean squared error

Regression trees
• Leaf node predicts average values of
training instances reaching that node
• Easy to interpret

Building the tree

• Splitting criterion: standard deviation
reduction
– minimizing intra-subset variation
• Termination criteria (important when
building trees for numeric prediction):
– Standard deviation becomes smaller than
certain fraction of s. d. for full training set
(e.g., 5%)
– Too few instances remain (e.g., less than
four)
• Pruning criterion: based on numeric error
measure

Decision trees and other classification algorithms can be adapted for regression
problems through a technique called discretization. In this context, discretization
involves partitioning the target variable (continuous in regression) into a set of distinct
intervals or ranges. Once the target variable is discretized, classification algorithms can
be applied to predict the interval to which a new instance belongs, and the prediction
for the new instance is then based on the interval's midpoint.

Here's a high-level overview of how decision trees and other classification algorithms
can be used for regression problems through discretization:

1. **Discretization**: The continuous target variable in the regression problem is

divided into a set of intervals or ranges. This can be achieved using various discretization
techniques such as equal-width binning, equal-frequency binning, or more advanced
methods like decision tree-based discretization.

2. **Training**: A classification algorithm, such as a decision tree, is trained on the

discretized target variable and the corresponding feature values.

3. **Prediction**: When making predictions for new instances, the trained model
assigns the new instance to the appropriate interval based on its feature values. The
prediction for the new instance is then calculated as the midpoint of the interval to
which it belongs.

4. **Evaluation**: The performance of the model is evaluated using regression metrics

such as mean squared error (MSE) or R-squared, which assess the accuracy of the
predictions made by the classification algorithm on the discretized target variable.

It's important to note that while this approach allows classification algorithms to be
used for regression problems, it may not capture the nuances of the original continuous
target variable as effectively as dedicated regression algorithms. Additionally, the choice
of discretization method and the number of intervals can impact the model's
performance.

In summary, decision trees and other classification algorithms can be adapted for
regression problems by discretizing the continuous target variable into intervals and
using classification techniques to predict the interval to which new instances belong,
with the prediction based on the interval's midpoint ,

Topic 5: Support Vector Machines

Support Vector Machines (SVM) are algorithms used for supervised learning and are particularly
effective for classification tasks. The primary goal of SVM is to find the optimal hyperplane that
best separates data points belonging to different classes. for learning linear classifiers
(for binary classification)
Find a linear hyperplane (decision boundary)
that will separate the data

Here's a brief explanation of how SVM works:

1. **Optimal Hyperplane**: SVM aims to find the hyperplane that maximizes the margin, i.e.,
the distance between the hyperplane and the nearest data points from each class. This
hyperplane is the one that best separates the classes and is known as the maximum margin
hyperplane.

2. **Support Vectors**: The data points that are closest to the hyperplane are called support
vectors. These support vectors are crucial in defining the hyperplane and are used to determine
the optimal decision boundary.

3. **Kernel Trick**: SVM can handle non-linear decision boundaries by mapping the input data
into a higher-dimensional space using a kernel function. This allows SVM to find a linear
hyperplane in the transformed space, effectively creating non-linear decision boundaries in the
original space.

Advantages of Support Vector Machines:

- Effective in high-dimensional spaces: SVM is effective even in cases where the number of
dimensions is greater than the number of samples.
- Versatile: SVM can be used for both classification and regression tasks.
- Robust to overfitting: SVM has a regularization parameter that helps avoid overfitting and
generalizes well to new data.

1. **Maximizes Margin - Robustness:**

- SVM aims to find a hyperplane that maximizes the margin, the distance between the
decision boundary and the nearest data points of each class.
- Maximizing the margin leads to a more robust model because it focuses on finding a decision
boundary that is less likely to be influenced by noise or outliers in the training data.
2. **Support for Kernels - Computable Approach for Nonlinear Problems:**
- SVMs can handle nonlinear relationships between input features and output by using kernel
functions.
- Kernels allow the SVM to implicitly map input data into a higher-dimensional space where a
linear decision boundary can be established.
- This ability to handle nonlinearities makes SVMs versatile for a wide range of complex
problems.

3. **Regularization Parameter to Reduce Overfitting:**

- SVM includes a regularization parameter (often denoted as C) that controls the trade-off
between achieving a smooth decision boundary and accurately classifying training data.
- By adjusting the regularization parameter, users can control the balance between fitting the
training data closely and preventing overfitting, which occurs when the model captures noise
instead of the underlying pattern.

4. **Kernel Engineering for Expert Knowledge:**

- The choice of a kernel function in SVM allows the incorporation of expert knowledge about
the problem at hand.
- Users can design custom kernels that capture specific domain knowledge, making SVMs
flexible in accommodating prior information and enhancing performance in certain types of
problems.

5. **Convex Optimization Problem - No Local Minima:**

- SVM formulation leads to a convex optimization problem, which means that there is a single
global minimum and no local minima.
- This property ensures that the optimization process reliably converges to the best solution,
avoiding the problem of getting stuck in suboptimal solutions that may occur in non-convex
optimization problems.
- Efficient optimization methods, such as Sequential Minimal Optimization (SMO), can be
applied to find the optimal solution efficiently.

Limitations of Support Vector Machines:

- Computationally intensive: Training an SVM can be time-consuming, especially on large
datasets.
- Sensitivity to the choice of kernel: The performance of SVM can be highly dependent on the
choice of the kernel function.
- Lack of transparency: The decision function of SVM is not easy to interpret, which can be a
drawback in some applications.

1. **Choosing Kernel Function:**

- One disadvantage of SVMs is the challenge of selecting an appropriate kernel function.
- The performance of an SVM can be sensitive to the choice of kernel, and it may not be clear
which kernel is best suited for a particular problem.
 - Poor selection of the kernel can lead to suboptimal results, making it crucial to have domain
knowledge or conduct extensive experimentation.

2. **Hyperparameter Selection for Avoiding Overfitting:**

- SVMs have hyperparameters, such as the regularization parameter (C), that need to be tuned
properly to achieve good generalization performance.
- Selecting the right hyperparameters can be challenging, and improper tuning may result in
overfitting or underfitting.
- Grid search or other optimization techniques are often employed to find the optimal
combination of hyperparameters, which can be computationally expensive.

3. **High Algorithmic Complexity and Memory Requirements:**

- SVMs, especially in their standard formulation, involve solving a quadratic programming
problem.
- The algorithmic complexity of solving the optimization problem is cubic in the number of
training examples, making it computationally expensive for large datasets.
- Memory requirements can also be substantial, particularly in tasks with a large number of
features or data points, which may limit the scalability of SVMs for certain large-scale
applications.

While SVMs offer powerful capabilities, addressing these challenges involves careful
consideration of kernel selection, hyperparameter tuning, and scalability concerns, making
them less straightforward to use in certain situations compared to simpler models.

In summary, Support Vector Machines are powerful algorithms for classification tasks,
particularly in high-dimensional spaces. They are effective in finding complex decision
boundaries and are robust to overfitting. However, they can be computationally intensive and
sensitive to the choice of kernel, and their decision function may lack transparency in some
cases.

The kernel trick in Support Vector Machines (SVMs) allows us to implicitly map the input data
into a higher-dimensional space without actually calculating the new attributes explicitly. This is
achieved by using a kernel function, which computes the dot product of the mapped data points
in the higher-dimensional space without explicitly transforming the data.

The key idea is that instead of working directly with the transformed feature space, we can
operate in the original input space by using the kernel function to compute the dot products
between the data points as if they were in the higher-dimensional space. This allows SVMs to
efficiently handle nonlinear relationships between the input features, as the kernel function
effectively captures the similarity between data points in the transformed space.

By using the kernel trick, SVMs can effectively model complex decision boundaries and handle
nonlinear classification problems without the need to explicitly compute the transformed
feature vectors, making the approach computationally efficient and versatile.
How to select kernel for SVM?
– Prior, expert knowledge
(automatic selection, tuning: very easy to
overfit!)

Handling noise:
1. **Assumption of Separability:**
- SVMs are originally designed with the assumption that the data is separable either in
the original feature space or in a transformed space through the use of kernel functions.
- The presence of noisy instances in the data, especially outliers or mislabeled points,
can significantly impact the performance of SVMs, as they may lead to suboptimal
decision boundaries.

2. **Applying SVMs to Noisy Data with Regularization (C parameter):**

- To address the impact of noisy data, SVMs introduce a regularization parameter,
often denoted as C.
- The regularization parameter C controls the trade-off between fitting the training data
closely and maintaining a smooth decision boundary.
- It bounds the influence of individual training instances on the decision boundary,
preventing outliers from having an excessively large impact.

3. **Influence Bounding via Constraint (0 <= αi <= C):**

- The regularization parameter is implemented through constraints on the Lagrange
multipliers (αi) associated with each training instance in the SVM optimization problem.
- The constraints ensure that the values of αi are within the range of 0 to C, limiting
the influence of any single training instance on the positioning of the decision boundary.

4. **Quadratic Optimization Problem:**

- Despite the introduction of the regularization parameter, solving the SVM
optimization problem remains a quadratic programming task.
- This implies that the algorithmic complexity is related to the square of the number of
training instances, and addressing noise in large datasets can be computationally
demanding.

5. **Determining C through Experimentation:**

- The value of the regularization parameter C needs to be determined through
experimentation, as there is no universal rule for setting its value.
- Users often perform grid search or other optimization techniques to find the optimal
value of C that balances the need for fitting the data with the goal of preventing
overfitting to noisy instances.

In summary, while SVMs can be applied to noisy data by introducing a regularization

parameter, users must carefully choose and experiment with the value of C to strike the
right balance between fitting the data and resisting the influence of outliers.
Applications: Machine vision: e.g., face identification
• Handwritten digit recognition: USPS data
• Bioinformatics: e.g., prediction of protein secondary structure
• Text classification

Support vector regression: Maximum margin hyperplane only applies to

classification
• However, idea of support vectors and kernel functions
can be used for regression
• Basic method same as in linear regression: want to minimize error. Differences
– ignore errors smaller than ε (user specified parameter)
– use absolute error instead of squared error
– simultaneously aim to maximize flatness of function (=avoid overfitting)

Topic 6: Bayesian networks, Naïve Bayes

“Opposite” of 1R: use all attributes
Naive Bayes is a simple yet powerful probabilistic classifier based on Bayes' theorem with the
"naive" assumption of feature independence.

How Naive Bayes Works:

1. **Bayes' Theorem**: Naive Bayes is based on Bayes' theorem, which describes the
probability of a hypothesis given the evidence. In the context of classification, it calculates the
probability of a class given the features.

2. **Feature Independence**: The "naive" assumption in Naive Bayes is that the features are
conditionally independent given the class. This means that the presence of a particular feature
in a class is independent of the presence of other features. Although assumption almost
never (entirely) correct, this scheme works quite often well in practice!

3. **Classification**: To classify a new instance, Naive Bayes calculates the probability of each
class given the features using Bayes' theorem and the naive assumption. The class with the
highest probability is assigned to the instance.

Advantages of Naive Bayes:

- **Simple and Fast**: Naive Bayes is simple to implement and computationally efficient,
making it suitable for large datasets and real-time applications.
- **Effective with High-Dimensional Data**: It performs well even with high-dimensional data
and a large number of features.
- **Robust to Irrelevant Features**: It is robust to irrelevant features and can handle noisy data
effectively.
- **Probabilistic Predictions**: Naive Bayes provides probabilistic predictions, allowing for
uncertainty estimation.

Limitations of Naive Bayes:

- **Strong Independence Assumption**: The assumption of feature independence may not
hold in many real-world datasets, which can lead to suboptimal performance.
- **Sensitivity to Outliers**: Naive Bayes can be sensitive to outliers and extreme values in the
data.
- **Inability to Learn Interactions**: It cannot capture interactions between features, as it
assumes independence.

Assumptions of Naive Bayes:

- **Feature Independence**: The main assumption is that the features are conditionally
independent given the class.
- **Class-Conditional Distributions**: It assumes the class-conditional distributions of features.

In summary, Naive Bayes is a simple and efficient classifier that makes the strong assumption of
feature independence. While it has advantages such as simplicity and speed, it also has
limitations related to its independence assumption and inability to capture feature interactions.
Understanding these characteristics is important when considering the use of Naive Bayes for
classification tasks

maximum likelihood estimation (MLE) of parameters given observations. MLE

attempts to find the parameter values that maximize the likelihood function, given the
observations
• Naive Bayes: learning: estimating parameter values of the model given data, based on
the MLE

The zero-frequency problem, also known as the zero-count problem, occurs in the context of
probabilistic classifiers, such as Naive Bayes, when an attribute value in the test data has not
been seen in the training data for a particular class. This leads to a situation where the conditional
probability of that attribute value given the class becomes zero, which in turn affects the
posterior probability calculation.

The zero-frequency problem can be problematic because a zero probability for an attribute value
given a class would cause the posterior probability for that class to also be zero, regardless of the
other attribute values. This can lead to incorrect classifications and loss of predictive power.

To address the zero-frequency problem, a common approach is to use a technique called Laplace
smoothing, also known as additive smoothing or Lidstone smoothing. The idea behind Laplace
smoothing is to add a small, non-zero value to the count of each attribute value for each class
during the probability estimation process. This ensures that no probability estimate is zero, and it
prevents the posterior probability from being zero when a particular attribute value has not been
seen in the training data for a class.
Mathematically, the Laplace smoothing adjustment can be represented as follows:
\[ P(x_i | y) = \frac{N_{yi} + 1}{N_y + d} \]
Where:
- \( N_{yi} \) is the count of instances with attribute value \( x_i \) and class \( y \) in the training
data.
- \( N_y \) is the total count of instances in class \( y \) in the training data.
- \( d \) is the number of possible values for the attribute.

By adding 1 to the numerator and \( d \) to the denominator, Laplace smoothing ensures that even
if an attribute value has not been seen for a particular class, it still has a non-zero probability
estimate. This helps to mitigate the zero-frequency problem and improves the robustness of the
probabilistic classifier, such as Naive Bayes, when dealing with unseen attribute values in the
test data.

In summary, the zero-frequency problem arises when an attribute value in the test data has not
been seen in the training data for a particular class, leading to zero probabilities and affecting the
posterior probability calculation. Laplace smoothing is a common technique used to address this
problem by adding a small value to the count of each attribute value for each class during
probability estimation, ensuring non-zero probability estimates and improving the classifier's
performance

Naïve Bayes works surprisingly well (even if independence assumption is clearly

violated)
• Too many redundant attributes (e.g., identical attributes) will cause problems
• Irrelevant attributes are OK, they are filtered out
• Fast to train, fast to classify
– even for large number of attributes
– incremental model building possible (streamed data)
• Outputs probability (?) distribution, not just a single class (+)
• (Does not learn training data perfectly (+/-))
• Typical application: text/spam detection

Decision boundary

Bayesian networks
A Bayesian network, also known as a belief network or a Bayes network, is a graphical model
used to represent probabilistic relationships among a set of variables. It is a powerful tool for
reasoning under uncertainty and has applications in various fields such as machine learning,
artificial intelligence, and healthcare.

Key Components of a Bayesian Network:

1. Nodes: In a Bayesian network, the variables of interest are represented as nodes. Each node
corresponds to a random variable and represents a specific aspect of the problem being
modeled.

2. Edges: The relationships between the variables are represented by directed edges between
the nodes. These edges indicate the probabilistic dependencies between the variables. The
direction of the edges shows the direction of the influence or causal relationship between the
variables.

3. Conditional Probability Distributions: Each node in a Bayesian network is associated with a

conditional probability distribution that quantifies the probabilistic relationship between that
node and its parent nodes. These distributions capture the likelihood of different values for a
node given the values of its parent nodes.

4. Directed Acyclic Graph (DAG): The graphical structure of a Bayesian network forms a directed
acyclic graph (DAG), meaning that the edges do not form any cycles. This acyclic property is
essential for the proper interpretation of conditional independence relationships among the
variables.

By utilizing these key components, Bayesian networks provide a compact and intuitive way to
represent complex probabilistic relationships and facilitate efficient probabilistic inference and
decision-making.

Bayes' theorem is a fundamental concept in probability theory that describes how to update the
probability of a hypothesis based on new evidence. In simple terms, it provides a way to revise
or update the probability of an event occurring given new information or evidence.

Mathematically, Bayes' theorem can be expressed as:

\[ P(A|B) = \frac{P(B|A) \times P(A)}{P(B)} \]

Where:
- \( P(A|B) \) is the probability of event A occurring given that event B has occurred.
- \( P(B|A) \) is the probability of event B occurring given that event A has occurred.
- \( P(A) \) and \( P(B) \) are the probabilities of events A and B occurring independently.
In essence, Bayes' theorem allows us to update our belief in the likelihood of an event (A) based
on the occurrence of another event (B), taking into account our prior knowledge of the
situation.

In the context of Bayesian networks, entropy is a fundamental concept used to quantify the
uncertainty associated with random variables or the amount of information contained in a
variable. Entropy plays a crucial role in understanding the probabilistic relationships and making
decisions within Bayesian networks. Here's how entropy is used in the context of Bayesian
networks:

1. Measure of Uncertainty: Entropy is used to measure the uncertainty or randomness

associated with the variables in a Bayesian network. It provides a quantitative measure of the
amount of information or surprise associated with the possible outcomes of a random variable.

2. Node Importance: In a Bayesian network, the entropy of a node (random variable) represents
the amount of uncertainty or information content associated with that variable. Nodes with
higher entropy indicate greater uncertainty, while nodes with lower entropy indicate more
predictable outcomes.

3. Information Gain: Entropy is used to calculate the information gain when making decisions or
performing inference in a Bayesian network. By evaluating the change in entropy before and
after observing evidence, one can quantify the reduction in uncertainty and the amount of
information gained from the evidence.

4. Decision Making: Entropy is utilized in decision-making processes within Bayesian networks.

It helps in identifying the most informative variables for making decisions or performing further
observations, as well as in determining the optimal paths for probabilistic inference.
Overall, entropy serves as a valuable tool for understanding the uncertainty and information
content within Bayesian networks, aiding in probabilistic reasoning, decision-making, and the
efficient representation of complex probabilistic relationships.

d-separation, also known as dependency separation, is a criterion used to determine the

conditional independence relationships between variables in a Bayesian network. It is a
fundamental concept in probabilistic graphical models and plays a crucial role in probabilistic
inference and decision-making.

In a Bayesian network, d-separation is a graphical criterion that determines whether two sets of
variables are conditionally independent given a third set of variables. The criterion is based on
the concept of blocking paths between variables in the network.

A path between two variables is said to be blocked if it satisfies one of the following conditions:

1. The path contains a node that is in the conditioning set (the third set of variables) and is a
parent of a node in one of the two sets being compared.

2. The path contains a node that is in the conditioning set and is a parent of a node that has a
non-descendant in the other set being compared.

If all paths between two sets of variables are blocked, then the two sets are d-separated, and
they are conditionally independent given the conditioning set. On the other hand, if there exists
an unblocked path between the two sets, then they are not d-separated, and they are not
conditionally independent given the conditioning set.

D-separation is a powerful tool for probabilistic inference and decision-making in Bayesian

networks. It allows for efficient computation of conditional probabilities and facilitates the
identification of the most informative variables for decision-making.

Bayesian networks learn by inferring the structure and parameters of the network from
observed data. There are two main scenarios for learning Bayesian networks: known structure
with full observability and unknown structure with full observability.

In the case of known structure with full observability, the structure of the Bayesian network is
already specified, and the learning process focuses on estimating the parameters of the
network based on the available data. This typically involves using techniques such as maximum
likelihood estimation and preventing overfitting using methods like Akaike Information Criterion
(AIC) and Minimum Description Length (MDL).

In the case of unknown structure with full observability, the goal is to learn both the structure
and parameters of the Bayesian network from the data. This involves evaluating the goodness of
a given network, searching through the space of possible networks, and learning the network
structure based on the observed data. Techniques such as the K2 algorithm, Tree-Augmented
Naive Bayes (TAN), and the Superparent one-dependence estimator are commonly used for
learning the structure of Bayesian networks in this scenario.

Overall, the learning process for Bayesian networks involves inferring the network structure and
estimating the parameters based on the available data, and various statistical and
computational techniques are employed to achieve this.

K2 algorithm: The K2 algorithm is a popular algorithm for learning the structure of Bayesian
networks in the case of unknown structure with full observability. It is a greedy hill-climbing
algorithm that starts with a given ordering of nodes (attributes) and processes each node in
turn, greedily trying to add edges from previous nodes to the current node.

The algorithm works by computing the conditional mutual information between the current
node and its potential parents, and then selecting the set of parents that maximizes this
conditional mutual information. The algorithm then adds the selected parents to the current
node and moves on to the next node in the ordering.

The K2 algorithm continues this process until no further edges can be added to the network, at
which point it returns the learned network structure. The result of the algorithm can depend on
the initial ordering of nodes, so it is often run multiple times with different orderings to ensure a
good result.

Overall, the K2 algorithm is a simple and efficient approach for learning the structure of
Bayesian networks in the case of unknown structure with full observability, and it has been
shown to perform well in practice.

TAN (Tree-Augmented Naive Bayes) is a popular algorithm for learning the structure of Bayesian
networks in the case of unknown structure with full observability. It is an extension of the Naive
Bayes algorithm, which assumes that the attributes are conditionally independent given the
class variable.

TAN starts with the Naive Bayes structure and then considers adding a second parent to each
node (apart from the class node) to capture any dependencies between the attributes. The
algorithm selects the second parent for each node by computing the conditional mutual
information between the node and each potential parent, and then selecting the parent that
maximizes this conditional mutual information.

Once the second parents have been added to the network, TAN constructs a maximum
spanning tree over the nodes and their parents, where the edges in the tree represent the most
informative dependencies between the attributes. The resulting network structure is a tree-
augmented Naive Bayes model that captures both the conditional independence assumptions
of Naive Bayes and the informative dependencies between the attributes.
TAN is an efficient algorithm for learning the structure of Bayesian networks, and it has been
shown to perform well in practice.

Topic 7: Instance based learning

Instance-based learning is a type of machine learning approach where the model's predictions
are based on the instances or examples in the training dataset. Instead of explicitly learning a
generalizable model from the training data, instance-based learning stores the training
instances and makes predictions for new instances based on their similarity to the stored
examples.

The key components of instance-based learning include:

1. **Storage of Training Instances**: In instance-based learning, the training instances are

stored in memory and serve as the knowledge base for making predictions. These instances are
typically represented as feature vectors in a multidimensional space.

2. **Similarity Measure**: To make predictions for a new instance, a similarity measure is used
to compare the new instance with the stored training instances. Common similarity measures
include Euclidean distance, cosine similarity, or other distance metrics depending on the nature
of the data.

3. **Prediction**: Once the similarity between the new instance and the stored instances is
calculated, the model uses this information to make predictions. For example, in the case of
classification, the model may assign the new instance the same class label as the most similar
training instance.

Instance-based learning, often associated with the k-nearest neighbors (KNN) algorithm, is
particularly useful when the underlying relationship between the input features and the target
variable is complex and not easily captured by a parametric model. It is also well-suited for non-
linear relationships and can adapt to the local structure of the data.

However, instance-based learning has some limitations, including the need to store the entire
training dataset, which can be memory-intensive, and the computational cost of making
predictions for new instances, especially in high-dimensional spaces.

In summary, instance-based learning is a machine learning approach that makes predictions for
new instances based on their similarity to stored training instances, making it particularly useful
for complex, non-linear relationships in the data
Topic 8: Clustering
KMeans

K-means clustering is a popular unsupervised machine learning algorithm used for partitioning a
dataset into a set of K clusters. The algorithm works by iteratively assigning data points to the
nearest cluster centroid and then updating the cluster centroids based on the mean of the data
points assigned to each cluster. This process continues until the cluster assignments stabilize or
a convergence criterion is met.

K-means scaling refers to techniques aimed at improving the efficiency and scalability of the K-
means algorithm. One approach to scaling K-means is mini-batch K-means, which involves using
a randomly sampled subset of the data (mini-batch) to update the cluster centroids, reducing
the computational time while still producing quality clustering results.

Advantages of K-means clustering include its simplicity, efficiency, and effectiveness in

identifying clusters in large datasets. However, K-means has some limitations, such as its
sensitivity to the initial choice of cluster centroids, its tendency to converge to local optima, and
its assumption of spherical clusters, which may not be suitable for all types of data distributions.

In summary, K-means clustering is a widely used algorithm for partitioning data into clusters,
and scaling techniques such as mini-batch K-means can improve its efficiency and applicability
to large datasets. However, users should be mindful of its limitations and consider alternative
clustering algorithms for non-spherical or complex data distributions.

Mean Shift clustering

Mean shift clustering is a non-parametric clustering algorithm that identifies clusters in a
dataset by iteratively shifting the centroids of candidate clusters towards the densest regions of
the data distribution The algorithm works by defining a sliding window around each data point
and computing the mean of the data points within the window. The centroid of the window is
then shifted towards the mean, and the process is repeated until convergence. The algorithm
can be visualized as a sliding window moving towards the densest regions of the data
distribution, with the centroids of candidate clusters being attracted to these regions. The size
of the sliding window can be adjusted to control the granularity of the clustering, with smaller
windows producing more fine-grained clusters and larger windows producing more coarse-
grained clusters.
One advantage of mean shift clustering is that it does not require the number of clusters to be
specified in advance, as the algorithm automatically identifies the number of clusters based on
the density of the data distribution. Additionally, mean shift clustering can handle non-linear
and non-convex data distributions, making it suitable for a wide range of applications.
Overall, mean shift clustering is a powerful and flexible clustering algorithm that can identify
clusters in complex data distributions without requiring the number of clusters to be specified in
advance.

Density-based clustering is a type of clustering algorithm that identifies clusters based on the
density of data points in the feature space. Unlike K-means, which partitions the data into
spherical clusters, density-based clustering algorithms are capable of identifying clusters of
arbitrary shapes and sizes, making them suitable for a wide range of data distributions.

One of the most popular density-based clustering algorithms is DBSCAN (Density-Based Spatial
Clustering of Applications with Noise. DBSCAN works by grouping together data points that are
within a specified distance of each other and have a minimum number of neighbors within that
distance. This approach allows DBSCAN to identify clusters as regions of high density separated
by regions of low density, without requiring the number of clusters to be specified in advance.

One key difference between DBSCAN and K-means is that DBSCAN does not assume a fixed
number of clusters and can identify clusters of varying shapes and sizes based on the density of
the data distribution. Additionally, DBSCAN is robust to noise and outliers, as it classifies data
points that do not belong to any cluster as noise, rather than forcing them into a cluster as K-
means would.

In summary, density-based clustering algorithms such as DBSCAN operate by identifying clusters

based on the density of data points, allowing them to handle arbitrary cluster shapes and sizes,
automatically determine the number of clusters, and robustly handle noise and outliers in the
data.

Probability-based clustering, also known as probabilistic clustering, involves assigning data

points to clusters based on the probability of belonging to each cluster. One popular algorithm
for probability-based clustering is the Expectation-Maximization (EM) algorithm using Gaussian
Mixture Models (GMM).

Gaussian Mixture Models represent the probability distribution of the data as a mixture of
multiple Gaussian distributions, each associated with a different cluster. The EM algorithm
iteratively estimates the parameters of the Gaussian distributions and the cluster assignments
to maximize the likelihood of the observed data.

The EM algorithm works as follows:

1. Initialization: Initialize the parameters of the Gaussian distributions and the cluster
assignments.
2. E-step (Expectation step): Estimate the probability of each data point belonging to each
cluster based on the current parameters of the Gaussian distributions.
3. M-step (Maximization step): Update the parameters of the Gaussian distributions based on
the current cluster assignments.
4. Iteration: Repeat the E-step and M-step until convergence, where the parameters and cluster
assignments stabilize.

Advantages of GMM and the EM algorithm include their ability to model complex data
distributions, handle overlapping clusters, and provide soft assignments of data points to
clusters based on probabilities. GMM can also capture the underlying structure of the data
more flexibly than some other clustering algorithms.

However, GMM and the EM algorithm have some limitations, such as their sensitivity to the
initial parameter values, their computational complexity, and their potential to converge to local
optima. Additionally, GMM may not perform well with high-dimensional data due to the curse
of dimensionality.

In summary, probability-based clustering using GMM and the EM algorithm provides a flexible
and probabilistic approach to clustering, but it is important to consider its limitations and the
specific characteristics of the data when applying this method.

Hierarchical clustering is a type of clustering algorithm that builds a hierarchy of clusters, which
can be represented as a tree-like dendrogram. Agglomerative clustering is a specific approach to
hierarchical clustering that starts with each data point as a single cluster and then iteratively
merges the closest pairs of clusters until only one cluster remains.

Agglomerative clustering works as follows:

1. Initialization: Start with each data point as a single cluster.
2. Merge: Iteratively merge the closest pair of clusters based on a chosen distance metric, such
as Euclidean distance.
3. Update distances: Recompute the distances between the merged cluster and the remaining
clusters.
4. Repeat: Continue merging clusters and updating distances until all data points belong to a
single cluster.

Agglomerative clustering can be visualized as a bottom-up approach, where individual data

points are successively merged into larger clusters until the desired number of clusters is
obtained or a stopping criterion is met.

One advantage of agglomerative clustering is that it does not require the number of clusters to
be specified in advance, and it can produce a hierarchy of clusters that provides insights into the
relationships between data points at different scales.

In summary, hierarchical clustering, particularly agglomerative clustering, is a powerful

approach for identifying clusters and understanding the structure of the data through the
creation of a cluster hierarchy.
Semi-supervised learning is a type of machine learning that combines both labeled and
unlabeled data to improve the accuracy of the learning model. In semi-supervised learning, a
small portion of the data is labeled, while the majority of the data is unlabeled.

The labeled data is used to train a model, which is then used to make predictions on the
unlabeled data. The predictions on the unlabeled data are then used to improve the model,
which is iteratively refined until convergence.

Semi-supervised learning is particularly useful in situations where labeled data is scarce or

expensive to obtain, but unlabeled data is plentiful. By leveraging the unlabeled data, semi-
supervised learning can improve the accuracy of the model beyond what would be possible
with only the labeled data.

Semi-supervised learning can be applied to a wide range of machine learning tasks, including
classification, regression, and clustering. Some common techniques for semi-supervised
learning include self-training, co-training, and multi-view learning.

In summary, semi-supervised learning is a powerful approach for improving the accuracy of

machine learning models by leveraging both labeled and unlabeled data. It is particularly useful
in situations where labeled data is scarce or expensive to obtain.

Combining classification and clustering involves using clustering to group similar instances
together and then using classification to assign labels to the resulting clusters. This approach
can improve the accuracy of classification by reducing the complexity of the problem and
providing more representative training data.

The process of combining classification and clustering can be broken down into the following
steps:
1. Clustering: Use a clustering algorithm to group similar instances together based on their
feature similarity.
2. Cluster labeling: Assign a label to each cluster based on the majority class of the instances
within the cluster or using some other criterion.
3. Classification: Train a classification model on the labeled clusters, using the cluster labels as
the target variable.
4. Prediction: Use the trained classification model to predict the labels of new instances.

One advantage of combining classification and clustering is that it can reduce the complexity of
the classification problem by grouping similar instances together and treating them as a single
entity. This can improve the accuracy of the classification model by reducing the noise and
variability in the data.

Another advantage is that it can provide more representative training data for the classification
model. By using the cluster labels as the target variable, the classification model can learn from
a larger and more diverse set of instances, which can improve its generalization performance.
However, one limitation of this approach is that it assumes that the clusters are homogeneous
and that the instances within each cluster share the same label. This may not always be the
case, and the resulting classification model may be biased or inaccurate if the clusters are not
well-defined or if there is significant overlap between the clusters.

In summary, combining classification and clustering can be a powerful approach for improving
the accuracy of classification models by reducing the complexity of the problem and providing
more representative training data. However, it is important to carefully consider the
characteristics of the data and the assumptions underlying the clustering and classification
algorithms when applying this approach.

Topic 9: Data transformation, hyperparameter tuning

Data transformation
simple transformations that can be applied to time series data include shifting values from the
past/future, computing the difference (delta) between instances (i.e., "derivative"), and
normalizing uneven time steps by step size. Additionally, Fourier (& co) transformation can also
be used for time series data.
Standardization, as mentioned in ,[object Object], on ,[object Object],, is the process of
preparing the data for analysis by ensuring that it meets the assumptions of the statistical
methods. Many algorithms expect the data to have unit variance, zero mean, and sometimes a
normal distribution. Standardization involves scaling the values of different attributes, which is
important for various algorithms such as K-means, SVM, instance-based methods, and PCA. It
can be achieved through techniques like min-max scaling and robust scaling to handle outliers.
Attribute selection, as mentioned in ,[object Object], on ,[object Object],, refers to the process
of selecting the most relevant attributes or features from a dataset. This process is crucial in
machine learning and data mining as it can lead to better prediction performance, higher speed,
and more understandable models. Attribute selection can be done manually based on a deep
understanding of the learning problem and the meaning of attributes, or automatically using
learning scheme-independent (filter method) or learning scheme-dependent (wrapper method)
approaches. It plays a significant role in improving the efficiency and accuracy of machine
learning models by reducing the dimensionality of the data and selecting the most informative
features.
CFS stands for Correlation-based Feature Selection. As mentioned in ,[object Object], on ,[object
Object],, CFS is a feature selection method that aims to eliminate both redundant and irrelevant
attributes from a dataset. It identifies a subset of attributes that individually correlate well with
the class while having little intercorrelation. CFS is particularly useful for improving the
performance of machine learning models by selecting the most relevant and uncorrelated
features, thereby reducing the dimensionality of the dataset and enhancing the model's
predictive power.
Principal Component Analysis (PCA), as described in ,[object Object], on ,[object Object],, is a
statistical method used to transform a dataset into a new coordinate system, where the new
axes (principal components) are orthogonal and capture the maximum amount of variance in
the data. The main idea behind PCA is to find the directions in the original feature space that
preserve the distances between instances, allowing for dimensionality reduction while retaining
the most important information.

In practical terms, PCA involves finding the eigenvectors of the covariance matrix of the data
through diagonalization. These eigenvectors, sorted by their corresponding eigenvalues,
represent the principal components or new directions in the transformed space. PCA is
commonly used for dimensionality reduction, visualization, noise reduction, and feature
extraction in various fields such as image processing, pattern recognition, and data analysis.

Discretization, as mentioned in ,[object Object], on ,[object Object],, is the process of

transforming continuous variables into discrete or categorical variables. This process is often
used in machine learning and data mining to simplify the data and make it more manageable for
analysis. Discretization can be done in two main ways: supervised and unsupervised.

Supervised discretization involves using class labels to guide the discretization process. This can
be done by building a decision tree on the attribute being discretized and using a splitting
criterion such as entropy to determine the best intervals. Supervised discretization is often used
for classification tasks.

Unsupervised discretization, on the other hand, involves generating intervals without looking at
class labels. This can be done using strategies such as equal-interval binning or equal-frequency
binning. Unsupervised discretization is often used when clustering data.

Discretization can help to reduce the dimensionality of the data, improve the performance of
machine learning algorithms, and make the data more interpretable. However, it can also lead
to information loss and should be done carefully and with consideration of the specific problem
at hand.

Sampling, as described in ,[object Object], on ,[object Object],, is the process of selecting a

subset of instances from a larger dataset to use for training or testing a machine learning model.
Sampling is often used when the dataset is too large to fit into memory or when the cost of
collecting or labeling data is high.

There are several types of sampling methods, including:

1. Random sampling: selecting instances randomly from the dataset.

2. Stratified sampling: selecting instances based on their class distribution to ensure that the
sample is representative of the population.
3. Reservoir sampling: selecting a fixed-size sample from a stream of instances, where the size
of the sample is not known in advance.

4. Oversampling: increasing the number of instances in the minority class to balance the class
distribution.

5. Undersampling: reducing the number of instances in the majority class to balance the class
distribution.

Sampling can help to improve the efficiency and accuracy of machine learning models by
reducing the size of the dataset, balancing the class distribution, and reducing the impact of
noisy or irrelevant instances. However, it can also lead to biased or unrepresentative samples if
not done carefully and with consideration of the specific problem at hand.
ECOCs stands for Error-Correcting Output Codes, as mentioned in ,[object Object], on ,[object
Object],. ECOCs are a technique used in multiclass classification problems to improve the
accuracy of machine learning models. The idea behind ECOCs is to represent each class as a
unique binary code, where each bit in the code corresponds to a different classifier. Each
classifier is trained to distinguish between one class and all the other classes, and the final
prediction is made by combining the outputs of all the classifiers using the binary code.

ECOCs can help to improve the accuracy of machine learning models by reducing the impact of
misclassifications and errors. Even if several classifiers make a mistake, the prediction is still
correct, and classes are not mistaken for each other. ECOCs use code words that have a large
Hamming distance between any pair, which can correct up to (d – 1)/2 single-bit errors.

There are two criteria for designing ECOCs: row separation and column separation. Row
separation ensures that classes are not mistaken for each other, while column separation
ensures that base classifiers are not likely to make the same errors. ECOCs only work for
problems with more than three classes, and it is not possible to achieve both row and column
separation for three classes.

One-vs-rest (OvR) and one-vs-one (OvO) are two common strategies for multiclass classification
using binary classifiers, as mentioned in ,[object Object], on ,[object Object],.

In OvR, also known as one-vs-all, a separate binary classifier is trained for each class, where the
positive class is the class of interest, and the negative class is the union of all the other classes.
During testing, the classifier with the highest output is chosen as the predicted class. OvR is a
simple and efficient strategy that works well for problems with a large number of classes, but it
can suffer from imbalanced class distributions.

In OvO, also known as all-pairs, a separate binary classifier is trained for each pair of classes,
where the positive class is one of the two classes, and the negative class is the other class.
During testing, each classifier makes a prediction, and the class with the most votes is chosen as
the predicted class. OvO is a more complex strategy that requires training more classifiers, but it
can be more accurate and robust to imbalanced class distributions.

Both OvR and OvO have their advantages and disadvantages, and the choice of strategy
depends on the specific problem at hand. OvR is often used for problems with a large number
of classes, while OvO is often used for problems with a small number of classes.

Calibrating class probabilities, as mentioned in ,[object Object], on ,[object Object], and ,[object
Object], on pages 45-46, is the process of adjusting the predicted probabilities of a machine
learning model to better reflect the true probabilities of the classes. The predicted probabilities
of a model may not be well calibrated, meaning that they may be too optimistic or too
pessimistic, and may not accurately reflect the true probabilities of the classes.

Calibration is important in many applications, such as cost-sensitive prediction, where the cost
of misclassification may vary depending on the class and the predicted probability. In such
cases, accurate class probabilities are necessary to make informed decisions.

There are several methods for calibrating class probabilities, including:

1. Platt scaling: fitting a logistic regression model to the outputs of the original model and using
the sigmoid function to transform the outputs into probabilities.

2. Isotonic regression: fitting a non-parametric model to the outputs of the original model and
using a monotonic function to transform the outputs into probabilities.

3. Bayesian calibration: using a Bayesian approach to estimate the true probabilities of the
classes based on the predicted probabilities and the prior distribution.

Calibrating class probabilities can help to improve the accuracy and reliability of machine
learning models, especially in applications where accurate probabilities are important. However,
it can also be computationally expensive and may require additional data or assumptions about
the problem at hand.

Robust regression, as mentioned in ,[object Object], on pages 35-36, is a variation of linear

regression that is designed to be less sensitive to outliers and violations of the assumptions of
classical linear regression. In classical linear regression, the model is fitted to minimize the sum
of squared errors, which makes it sensitive to outliers and non-normal errors.

Robust regression addresses this issue by using alternative loss functions and estimation
methods that are less affected by outliers and non-normal errors. Some common approaches to
robust regression include:

1. Minimizing absolute error (L1 norm) instead of squared error (L2 norm): This approach,
known as L1 regression or least absolute deviations, minimizes the sum of the absolute
differences between the observed and predicted values, rather than the squared differences.
This makes the model less sensitive to large errors and outliers.

2. Removing outliers: In some cases, outliers can be identified and removed from the dataset
before fitting the regression model. This can help to reduce the influence of outliers on the
model's parameters.

3. Minimizing the median of squares: Instead of minimizing the mean of squared errors, some
robust regression methods minimize the median of squared errors. This approach is less
affected by outliers in both the x and y directions.

Robust regression is particularly useful in situations where the data may contain outliers or
errors, and where the assumptions of classical linear regression may not hold. By using robust
regression techniques, it is possible to obtain more reliable and accurate estimates of the
relationships between variables, even in the presence of problematic data points.

Hyperparameter tuning is the process of selecting the best set of hyperparameters for a
machine learning algorithm to optimize its performance. Hyperparameters are configuration
settings for a model that are not learned from the data, such as learning rate, regularization
strength, or the number of hidden layers in a neural network. Tuning these hyperparameters is
crucial for achieving the best possible performance from a model.

On the other hand, model parameters are the variables that are learned from the training data
during the model fitting process. They are the internal variables that the model uses to make
predictions, such as weights in a neural network or coefficients in a linear regression model.

These concepts are essential for understanding the process of hyperparameter tuning and the
challenges involved in finding the best hyperparameter values for a given machine learning
model.
The evaluation framework for hyperparameter tuning typically involves techniques to assess the
performance of different hyperparameter configurations. Common components of the
evaluation framework include:

1. Three-way data split: The dataset is divided into three parts - training set, validation set, and
test set. The training set is used to train the model, the validation set is used to tune the
hyperparameters, and the test set is used to evaluate the final model performance.

2. Nested cross-validation: This technique involves performing cross-validation on the training

set to tune the hyperparameters and then using a separate validation set to assess the model's
performance. Nested cross-validation helps in reducing the risk of overfitting during
hyperparameter tuning.

3. Cross-validation for tuning: Cross-validation is often used to assess the performance of

different hyperparameter configurations. It involves splitting the training data into multiple
folds, training the model on a subset of the data, and evaluating it on the remaining data. This
process is repeated for each hyperparameter configuration to estimate its performance.

These techniques are essential for rigorously evaluating the performance of different
hyperparameter configurations and selecting the best set of hyperparameters for a given
machine learning model.
There are several methods for hyperparameter tuning, including:

1. Manual selection: This involves manually selecting hyperparameters based on prior

knowledge or experience. While this method is simple, it can be time-consuming and may not
always lead to the best performance.

2. Grid search: This involves defining a grid of hyperparameter values and searching through this
grid to find the best combination of hyperparameters. While grid search is straightforward and
exhaustive, it can be computationally expensive and impractical when dealing with a large
number of hyperparameters or a large search space.

3. Random search: This involves randomly sampling hyperparameters from a defined search
space. While random search is less computationally expensive than grid search, it may not
always find the optimal solution and may require more iterations to converge.

4. Bayesian optimization: This involves building a probabilistic model of the objective function
and using it to guide the search for the best hyperparameters. Bayesian optimization is
computationally efficient and can handle noisy or non-convex objective functions, but it
requires more expertise to implement.

5. Evolutionary algorithms: This involves using genetic algorithms or other evolutionary

techniques to search for the best hyperparameters. Evolutionary algorithms can handle complex
search spaces and can find good solutions quickly, but they may require more computational
resources and can be difficult to interpret.

These methods have their own advantages and disadvantages, and the choice of method
depends on the specific problem and available resources.

Topic 10: Ensemble learning

Ensemble learning is a machine learning technique that involves combining multiple individual
models to create a stronger, more accurate predictive model. The basic idea is that by
aggregating the predictions of multiple models, the overall predictive performance can be
improved. •Advantage: – often very good predictive performance
•Disadvantage: – interpretation of model ? (black box...)
There are several different types of ensemble learning models, each with its own approach to
combining individual models:
1. Bagging (Bootstrap Aggregating): In bagging, multiple instances of the same learning
algorithm are trained on different subsets of the training data. The final prediction is then made
by averaging the predictions of all the individual models. This approach helps to reduce variance
and overfitting in the final model.

• Bootstrap Sampling: Multiple random subsets of the training data are created through
bootstrap sampling, where each subset is of the same size as the original training set but
contains random samples with replacement.

• Model Training: A base learning algorithm, such as decision trees or neural networks, is
trained on each bootstrap sample to create multiple diverse models.

• Aggregation: The predictions from each model are combined, often through averaging or
voting, to produce the final prediction.

Advantages of Bagging:
1. Reduction of Variance: By training models on different subsets of the data, bagging helps
reduce the variance of the final model, making it less sensitive to small changes in the training
data and less prone to overfitting.

2. Improved Generalization: Bagging can lead to improved generalization performance by

combining the predictions of multiple models, especially when the base models are diverse and
complementary.

3. Stability: Bagging can increase the stability of the model by reducing the impact of outliers or
noisy data points in the training set.

Limitations of Bagging:
1. Interpretability: The final model produced by bagging, especially when using complex base
models, may be less interpretable compared to a single model.

2. Computational Cost: Training multiple models in parallel can be computationally expensive,

especially for large datasets or complex learning algorithms.

3. Potential Overfitting: While bagging aims to reduce overfitting, it is still possible for the base
models to overfit the training data, especially if the base learning algorithm is prone to high
variance.

In summary, bagging is a powerful ensemble learning technique that can improve the stability
and generalization performance of machine learning models, especially when used with diverse
base models. However, it is important to consider the trade-offs in terms of interpretability and
computational cost when applying bagging to a given machine learning problem.
2. Boosting: Boosting is an iterative ensemble learning technique where a sequence of models
are trained, with each new model focusing on the instances that were misclassified by the
previous models. The final prediction is made by combining the predictions of all the models,
often using a weighted average. This approach aims to improve the overall predictive
performance by focusing on the instances that are more challenging to classify.
Boosting: explicitely seeks models
that complement one another
• Works well with weak models
Similar to Bagging
– Voting (classification), averaging (numeric
prediction)
– Models of the same type (e.g. decision trees)
• Different from bagging
– Iterative, models not built separately
– New model is encouraged to become expert for
instances classified incorrectly by earlier models
• Intuitive justification: models should be experts that
complement each other
– Weighted voting!
• There are several variants of this algorithm: AdaBoost (Additive model, particular loss
function), GradientBoos t AdaBoost: adjusting weights of data
points...
• Difference between the prediction and
the ground truth.
– Optimization problem
• on suitable cost function
• over function space
• Gradiant methods
– Iterative, greedy
– by choosing function pointing in negative
gradient direction
, XGBoost Decision tree ensembles
• Reminder:
• Regularisation!
– Formalizes complexity of tree classifiers
overfitting reduced!!
XGBoost, which stands for eXtreme Gradient Boosting, is a powerful and popular machine
learning algorithm known for its numerous advantages. Some of the key advantages of XGBoost
include:

1. Custom Optimization ives and Evaluation Criteria: XGBoost allows for the customization of
optimization ives and evaluation criteria, enabling the algorithm to be tailored to specific
problem domains and performance metrics.
2. Handling Missing Values: XGBoost has built-in capabilities to handle missing values within the
dataset, reducing the need for extensive data preprocessing and imputation techniques.

3. Built-in Cross Validation: The algorithm includes built-in cross-validation functionality,

simplifying the process of evaluating model performance and tuning hyperparameters.

4. Continuation of Existing Model: XGBoost supports the ability to continue training an existing
model, allowing for incremental learning and the incorporation of new data without starting the
training process from scratch.

5. Parallel Processing: XGBoost is designed for efficient parallel processing, enabling faster
model training through the utilization of multiple CPU cores.

6. Support for Sparse Matrices: XGBoost incorporates algorithms that are aware of sparse
matrices, optimizing performance when dealing with high-dimensional and sparse datasets.

7. Improved Data Structures for Processor Cache Utilization: XGBoost utilizes enhanced data
structures to maximize processor cache utilization, leading to improved computational
efficiency and faster training times.

8. Multicore Processing Support: The algorithm provides better support for multicore
processing, further reducing overall training time by leveraging the computational power of
multiple CPU cores.

These advantages make XGBoost a popular choice for various machine learning tasks, including
classification, regression, and ranking problems, and contribute to its reputation as a high-
performance and versatile algorithm in the machine learning community
XGBoost
– Easier to train
– Less computational resources
– Better if categorical + numeric features
• Deep learning
– Image Recognition, Computer Vision, Natural
Language Processing (some sort of structure,
space)

3. Random Forest: Random Forest is a specific type of ensemble learning model that uses a
collection of decision trees. Each tree is trained on a random subset of the features and the final
prediction is made by aggregating the predictions of all the trees. Random Forest is known for
its ability to handle high-dimensional data and reduce overfitting.
Rotation forest: Bagging creates ensembles of accurate classifiers with
relatively low diversity
 Bootstrap sampling creates training sets with a
distribution that resembles the original data
 Randomness in the learning algorithm increases diversity
but sacrifices accuracy of individual ensemble members
 Accuracy-diversity dilemma
 Rotation forests have the goal of creating accurate and
diverse ensemble members
Rotation forest”
 Base classifier: decision tree  „forest”
 PCA is a simple rotation of the coordinate axes 
„rotation”
 Rotation forest has the potential to improve
on diversity significantly without
compromising the individual accuracy

4. Stacking: Stacking, also known as stacked generalization, involves training a meta-model that
combines the predictions of multiple base models. The base models can be of different types
and are trained on the same dataset. The meta-model then learns how to best combine the
predictions of the base models to make the final prediction.
Data used for training level-0 models
not to be used to train level-1 model! –
overfitting!
– Cross-validation-like scheme is
employed

4. Stacking: Stacking, also known as stacked generalization, involves training a meta-model that
combines the predictions of multiple base models. The base models can be of different types
and are trained on the same dataset. The meta-model then learns how to best combine the
predictions of the base models to make the final prediction.

1. Base Model Training: Multiple diverse base models are trained on the same dataset using
different learning algorithms or variations of the same algorithm.

2. Meta-Model Training: The predictions made by the base models are used as input features
for training a meta-model, which learns how to best combine the predictions of the base
models to make the final prediction.

3. Final Prediction: When making predictions on new data, the base models generate their
predictions, which are then used as input to the meta-model to produce the final prediction.

Advantages of Stacking:
1. Improved Predictive Performance: Stacking can lead to improved predictive performance by
leveraging the strengths of multiple diverse base models and learning how to best combine
their predictions.
2. Model Diversity: Stacking allows for the use of diverse base models, which can capture
different aspects of the data and improve the overall generalization performance.

3. Flexibility: Stacking is a flexible ensemble learning technique that can accommodate a wide
range of base models and can be customized to suit the specific characteristics of the dataset
and the problem at hand.

Limitations of Stacking:
1. Complexity: Stacking introduces additional complexity into the modeling process, as it
requires training multiple base models and a meta-model, as well as managing the integration
of predictions from the base models.

2. Data Leakage: There is a risk of data leakage when using the predictions of the base models
as input features for the meta-model, which can lead to overfitting if not properly managed.

3. Computational Cost: Training multiple base models and a meta-model can be computationally
expensive, especially for large datasets or complex learning algorithms.

In summary, stacking is a powerful ensemble learning technique that can lead to improved
predictive performance by combining the strengths of multiple base models. However, it is
important to consider the trade-offs in terms of complexity, data leakage, and computational
cost when applying stacking to a given machine learning problem.
Each of these ensemble learning models has its own strengths and weaknesses, and the choice
of model depends on the specific characteristics of the dataset and the problem at hand. By
leveraging the diversity of multiple models, ensemble learning can often lead to improved
predictive performance compared to using a single model.

Topic 11: Logistic regression, Perceptron, Winnow, Neural networks

Logistic regression is a statistical method used for modeling the probability of a binary outcome.
It is commonly employed in binary classification problems, where the goal is to predict the
probability of a certain event occurring. Here's a brief overview of logistic regression:

1. **Model Formulation**: In logistic regression, the relationship between the independent

variables and the log-odds of the dependent variable is modeled using the logistic function. The
logistic function, also known as the sigmoid function, maps any real-valued number to the range
[0, 1], making it suitable for modeling probabilities.

2. **Working Principle**: The logistic regression model calculates the probability that a given
input belongs to a certain category. It does this by taking a linear combination of the input
features and applying the logistic function to the result. The output of the logistic function
represents the probability of the input belonging to a particular class.

3. **Assumptions**:
 - **Linear Relationship**: Logistic regression assumes a linear relationship between the
independent variables and the log-odds of the dependent variable.
- **Independence of Predictors**: The independent variables should be independent of each
other to avoid multicollinearity, which can lead to unreliable estimates of the coefficients.
- **Large Sample Size**: Logistic regression typically requires a relatively large sample size to
produce stable estimates.

4. **Prediction**: Once the model is trained, it can be used to predict the probability of a new
input belonging to a particular class. The predicted class is then determined based on a chosen
threshold (commonly 0.5), where probabilities above the threshold are classified as one class,
and those below are classified as the other.

Logistic regression is widely used in various fields, including healthcare, marketing, and social
sciences, due to its interpretability and effectiveness in modeling binary outcomes.

Parameters are found from training data using maximum

likelihood
• Aim: choosing parameters to maximize probability of
training data („maximum likelihood”)
– Independence: probabilities multiplied
– Maximizing multiplied probabilities : maximizing sum of
logarithms of probabilities

Perceptron: The perceptron is a fundamental algorithm in the field of machine learning,

specifically in the realm of binary classification. Here's an overview of the perceptron
and how it works:

1. **Definition**: The perceptron is a type of linear classifier used for binary

classification tasks, where it aims to separate input data points into two classes by
finding an optimal decision boundary.

2. **Working Principle**:
- **Input Features**: The perceptron takes a set of input features (attributes) and
assigns a weight to each feature.
- **Activation Function**: It computes the weighted sum of the input features and
applies an activation function (commonly a step function or a sign function) to produce
the output.
- **Learning**: During the learning phase, the perceptron adjusts the weights based
on the errors made in the previous iterations. This process continues until the model
converges or a predefined number of iterations is reached.

3. **Assumptions**:
- **Linear Separability**: The perceptron assumes that the input data is linearly
separable, meaning that it can be divided into two classes by a linear decision
boundary.
 - **Convergence**: The perceptron algorithm assumes that the training data is linearly
separable, and if this is the case, the algorithm is guaranteed to converge and find a
solution.

4. **Prediction**: Once trained, the perceptron can be used to predict the class of new
input data points by applying the learned weights to the input features and determining
the output based on the activation function.

5. **Limitations**: One of the main limitations of the perceptron is its inability to handle
non-linearly separable data, which led to the development of more advanced algorithms
such as multi-layer perceptrons and support vector machines to address this issue.

The perceptron algorithm laid the foundation for neural network models and played a
significant role in the development of the field of artificial intelligence and machine
learning.
The key components of a perceptron include the input features, weights, a weighted
sum function, an activation function, and the output. The perceptron takes input values,
multiplies them by corresponding weights, calculates the weighted sum, applies an
activation function to the sum, and produces an output based on the result.

The weighted sum function can be represented as:

\[ z = w_1x_1 + w_2x_2 + ... + w_nx_n + b \]
where \( w_i \) are the weights, \( x_i \) are the input features, \( n \) is the number of
input features, and \( b \) is the bias.

The activation function introduces non-linearity and determines the output of the
perceptron based on the weighted sum. Common activation functions include step
functions for traditional perceptrons and sigmoid functions for more advanced models.

The output of the perceptron is determined by the result of the activation function and
can be used for making predictions or further processing within a neural network.

Winnow is a binary classification algorithm that operates on the principle of mistake-

driven learning and is designed to efficiently handle high-dimensional feature spaces.

1. **Binary Classification**: Winnow is primarily used for binary classification tasks,

where the goal is to classify input instances into one of two categories.

2. **Mistake-Driven Learning**: Winnow employs a mistake-driven learning approach,

where it adjusts its weights based on the mistakes made during the classification
process. It focuses on updating the weights of relevant features to improve classification
accuracy.

3. **User-Specific Threshold**: Unlike the perceptron algorithm, Winnow utilizes a user-

specific threshold for classification. This threshold determines the decision boundary for
classifying instances.
4. **Multiplicative Updates**: In contrast to the additive updates used in the perceptron
algorithm, Winnow employs multiplicative updates for adjusting the weights. This
approach allows the algorithm to focus on relevant features by increasing or decreasing
their weights based on classification errors.

5. **Learning Process**:
- **Initialization**: Initially, the algorithm sets the weights for all features to 1.
- **Classification**: For each input instance, Winnow computes the weighted sum of
the input features and compares it to the user-specific threshold to make a classification
decision.
- **Weight Updates**: If a misclassification occurs, the algorithm updates the weights
of the relevant features by multiplying them with a user-specified parameter (commonly
denoted as alpha) to increase or decrease their influence on future classifications.

6. **Prediction**: Once trained, Winnow can be used to predict the class of new input
instances by applying the learned weights to the input features and comparing the result
to the user-specific threshold.

Winnow's focus on relevant features and its ability to handle high-dimensional feature
spaces make it a valuable algorithm for certain classification tasks, particularly in
scenarios with large numbers of features.

Neural Network: A neural network is a computational model inspired by the structure and
functioning of the human brain. It consists of interconnected nodes, called neurons, organized
in layers. Neural networks are capable of learning from data and can be used for tasks such as
classification, regression, pattern recognition, and more.

The basic building block of a neural network is the perceptron, which takes multiple input
values, applies weights to these inputs, computes a weighted sum, and then applies an
activation function to produce an output. Multiple perceptrons are organized into layers, and
the output of one layer serves as the input to the next layer. The first layer is the input layer, the
last layer is the output layer, and any layers in between are called hidden layers.

Neural networks learn by adjusting the weights and biases of the connections between neurons
based on the input data and the desired output. This process is known as training, and it
typically involves an optimization algorithm such as gradient descent and a method for
propagating errors backward through the network, known as backpropagation.

During the training process, the network aims to minimize a cost function by iteratively
adjusting the weights and biases. Once trained, the neural network can make predictions or
perform tasks based on new input data.
Neural networks have the ability to learn complex patterns and relationships in data, making
them powerful tools for a wide range of applications in machine learning and artificial
intelligence.

A feedforward neural network is a type of neural network where the information flows in only
one direction, from the input layer to the output layer, without any feedback loops. In other
words, the output of one layer serves as the input to the next layer, and there are no
connections between neurons in the same layer or between neurons in adjacent layers.

The most common type of feedforward neural network is the multilayer perceptron (MLP),
which consists of an input layer, one or more hidden layers, and an output layer. Each layer is
composed of multiple neurons, and each neuron is connected to all the neurons in the previous
layer and all the neurons in the next layer.

The neurons in the hidden layers use activation functions to transform the input data into a
form that is more useful for the task at hand. The output layer produces the final output of the
network, which can be used for classification, regression, or other tasks.

Feedforward neural networks are trained using supervised learning, where the network is
presented with input data and the corresponding desired output, and the weights and biases of
the network are adjusted to minimize the difference between the predicted output and the
desired output. This process is typically done using an optimization algorithm such as gradient
descent and backpropagation.

Feedforward neural networks have been successfully applied to a wide range of tasks, including
image and speech recognition, natural language processing, and financial forecasting.

Gradient descent is an optimization algorithm used to train neural networks by minimizing a

cost function. The cost function measures the difference between the predicted output of the
network and the actual output, and the goal of training is to find the set of weights and biases
that minimize this difference.

The basic idea behind gradient descent is to iteratively adjust the weights and biases of the
network in the direction of the steepest descent of the cost function. This is done by computing
the gradient of the cost function with respect to the weights and biases, and then updating the
weights and biases in the opposite direction of the gradient.

The gradient is computed using the backpropagation algorithm, which propagates the error
from the output layer back through the network to the input layer, and computes the gradient
of the cost function with respect to each weight and bias in the network.

There are different variants of gradient descent, including batch gradient descent, stochastic
gradient descent, and mini-batch gradient descent. In batch gradient descent, the entire training
dataset is used to compute the gradient at each iteration, which can be computationally
expensive for large datasets. In stochastic gradient descent, only one training example is used to
compute the gradient at each iteration, which can be faster but may result in noisy updates.
Mini-batch gradient descent is a compromise between the two, where a small batch of training
examples is used to compute the gradient at each iteration.

Gradient descent is an iterative process, and the training process continues until the cost
function reaches a minimum or a predefined stopping criterion is met.

Backpropagation is a key algorithm for training neural networks. It is used to calculate the
gradient of the loss function with respect to the weights of the network, allowing for the
iterative adjustment of the weights to minimize the loss.

The backpropagation algorithm consists of two main phases: the forward pass and the
backward pass.

1. Forward Pass:
- During the forward pass, input data is fed into the network, and the network's predictions
are computed layer by layer, moving from the input layer to the output layer.
- The predictions are compared to the actual targets using a loss function, which measures the
difference between the predicted output and the true output.

2. Backward Pass:
- In the backward pass, the gradient of the loss function with respect to the weights of the
network is computed using the chain rule of calculus.
- The gradient is calculated by propagating the error backwards through the network, starting
from the output layer and moving towards the input layer. This process involves computing the
partial derivatives of the loss function with respect to the weights and biases of the network.

The calculated gradients are then used to update the weights and biases of the network in the
direction that minimizes the loss function, typically using an optimization algorithm such as
gradient descent.

Backpropagation allows the network to learn from its mistakes by adjusting the weights and
biases based on the computed gradients. This iterative process continues until the network's
predictions closely match the true targets, indicating that the network has learned to make
accurate predictions.

Backpropagation is a fundamental component of neural network learning, enabling the network

to adjust its parameters in response to training data and improve its performance over time.

A mini-batch is a subset of the training data used in the training of neural networks. Instead of
using the entire training dataset at once, mini-batches divide the data into smaller chunks,
allowing for more efficient computation during the training process.
Mini-batch training is a compromise between batch training (using the entire dataset for each
iteration) and stochastic training (using one data point at a time). By using mini-batches, the
training process can benefit from the advantages of both batch and stochastic training.

The main advantages of mini-batch training include:

1. Efficiency: Processing the entire training dataset at once can be computationally expensive,
especially for large datasets. Mini-batches allow for more efficient computation, as the
gradients are computed based on a smaller subset of the data.
2. Regularization: Mini-batch training introduces a form of noise into the gradient updates,
which can act as a form of regularization and help prevent overfitting.
3. Convergence: Mini-batch training can lead to faster convergence compared to stochastic
training, as it provides a balance between the stability of batch training and the rapid updates of
stochastic training.

During each training iteration, a mini-batch of data is fed into the network, and the gradients of
the loss function with respect to the weights and biases are computed based on the mini-batch.
The weights and biases are then updated using the computed gradients, and this process is
repeated for each mini-batch until the entire training dataset has been used.

The size of the mini-batch, known as the batch size, is a hyperparameter that can be tuned
based on the specific characteristics of the dataset and the computational resources available.

Advantages of Neural Networks:

1. Non-linearity: Neural networks can model complex non-linear relationships between inputs
and outputs, making them well-suited for tasks such as image and speech recognition, natural
language processing, and financial forecasting.
2. Adaptability: Neural networks can adapt to new data and learn from experience, making
them useful for tasks where the underlying patterns may change over time.
3. Parallel processing: Neural networks can perform many computations in parallel, making
them well-suited for tasks that require high computational power, such as image and speech
recognition.
4. Robustness: Neural networks can handle noisy and incomplete data, making them useful for
tasks where the data may be imperfect or incomplete.
5. Generalization: Neural networks can generalize well to new data, making them useful for
tasks where the goal is to make predictions on unseen data.

Limitations of Neural Networks:

1. Black box: Neural networks can be difficult to interpret, as the relationship between the
inputs and outputs is not always clear. This can make it difficult to understand how the network
is making its predictions.
2. Overfitting: Neural networks can be prone to overfitting, where the network becomes too
complex and starts to memorize the training data instead of learning the underlying patterns.
This can lead to poor performance on new data.
3. Computational complexity: Neural networks can be computationally expensive to train and
require large amounts of data and computational resources.
4. Hyperparameter tuning: Neural networks have many hyperparameters that need to be tuned,
such as the number of layers, the number of neurons per layer, the learning rate, and the
regularization parameters. Tuning these hyperparameters can be time-consuming and require
expertise.
5. Data requirements: Neural networks require large amounts of data to train effectively, and
the quality of the data can have a significant impact on the performance of the network.

Overall, neural networks have many advantages and are well-suited for a wide range of tasks.
However, they also have limitations that need to be considered when deciding whether to use
them for a particular task.

Topic 12: Performance evaluation – measures

Model evaluation can be conducted through the following steps:

1. Split the data: Split the available data into training, validation, and test sets ,[ ],. The training
set is used to train the model, the validation set is used to select the best model and tune
hyperparameters, and the test set is used to evaluate the final model's performance.

2. Train the model: Train the model on the training set using a chosen algorithm and
hyperparameters ,[ ],.

3. Evaluate the model: Evaluate the model's performance on the validation set using
appropriate performance measures, such as accuracy, precision, recall, F1-score, or AUC

4. Tune the model: If the model's performance is not satisfactory, adjust the hyperparameters
and repeat steps 2 and 3 until the desired performance is achieved

5. Test the model: Once the final model is selected, evaluate its performance on the test set
using the same performance measures as in step 3

6. Interpret the results: Analyze the results and draw conclusions about the model's
performance and its suitability for the given task ,[ ],.

It's important to note that model evaluation should be conducted with caution to avoid
overfitting and data leakage. Additionally, the choice of performance measures and evaluation
methods should be appropriate for the specific type of machine learning problem and the goals
of the analysis
Models can be selected and/or tuned through the following steps:

1. Model Selection:
 - Choose a set of candidate models/algorithms that are suitable for the given task and dataset
,[ ],.
- Train each model using the training set and evaluate their performance using the validation
set ,[ ],.
- Select the best-performing model based on the evaluation results ,[ ],.

2. Hyperparameter Tuning:
- For the selected model, tune its hyperparameters to optimize its performance ,[ ],.
- Use techniques such as grid search, random search, or Bayesian optimization to
systematically explore the hyperparameter space and find the best combination ,[ ],.
- Evaluate the performance of the tuned model using the validation set and select the optimal
hyperparameters ,[ ],.

3. Data Preprocessing:
- Preprocess the data as needed, such as handling missing values, scaling features, encoding
categorical variables, and feature engineering ,[ ],.
- The preprocessing steps should be applied consistently to the training, validation, and test
sets to avoid data leakage ,[ ],.

4. Postprocessing:
- Apply any necessary postprocessing steps to the model's outputs, such as thresholding
probabilities, ensembling multiple models, or calibrating predictions ,[ ],.

It's important to conduct model selection and tuning in a principled manner, avoiding data
leakage and overfitting. Additionally, the performance of the final tuned model should be
assessed using the test set to ensure its generalization to new, unseen data ,[ ],.

Various measures can be used for performance evaluation, depending on the specific type of
machine learning problem and the goals of the analysis. Some common measures include:

1. Classification Problems:
- Error rate: Proportion of errors made over the whole set of instances. Success rate: (TP +
TN)/(TP+FP+TN+FN)
•
Error rate: (FP + FN)//(TP+FP+TN+FN)

- 0-1 loss function: Measures the success rate of classification

- F-measure: Harmonic mean of precision and recall
- Area under the ROC curve (AUC): Measures the probability that a randomly chosen positive
instance is ranked above a randomly chosen negative one
- Confusion matrix: Provides a detailed breakdown of correct and incorrect predictions for
each class in a multi-class problem
2. Regression Problems:
- Squared error: Commonly used for numeric prediction
- Root mean-squared error: Measures the average of the squares of the errors
- Mean absolute error: Measures the average of the absolute errors

3. Probability Estimation:
- Log loss: Measures the performance of a classifier that outputs a probability
- Average precision at different recall levels: Evaluates the accuracy of probability estimates
It's important to consider the specific characteristics of the problem and the goals of the
analysis when selecting the appropriate performance measure

**Precision:**
- Precision is a measure of the accuracy of the positive predictions made by a classification or
clustering model.
- It is calculated as the ratio of true positive predictions to the total number of positive
predictions (true positives + false positives).
- Precision gives an indication of how reliable the model is when it predicts a positive outcome.

**Recall (Sensitivity):**
- Recall, also known as sensitivity or true positive rate, assesses the ability of a model to capture
all the relevant instances of a positive class.
- It is calculated as the ratio of true positive predictions to the total number of actual positive
instances (true positives + false negatives).
- Recall helps identify how well the model avoids missing positive instances.

**Specificity:**
- Specificity is a measure of the accuracy of the negative predictions made by a classification or
clustering model.
- It is calculated as the ratio of true negative predictions to the total number of actual negative
instances (true negatives + false positives).
- Specificity indicates how well the model avoids misclassifying negative instances.

**Sensitivity (Recall):**
- Sensitivity, also known as recall, measures the ability of a model to correctly identify positive
instances.
- It is calculated as the ratio of true positive predictions to the total number of actual positive
instances (true positives + false negatives).
- Sensitivity is particularly useful in scenarios where missing positive instances is a critical
concern, such as in medical diagnoses.

Cross entropy and Brier score are both metrics used to evaluate the performance of
probabilistic predictions, such as those generated by classification models. Here's a brief
explanation of each:
1. Cross Entropy:
- Cross entropy is a measure of the difference between two probability distributions. In the
context of classification models, it quantifies the difference between the predicted probabilities
assigned to the true class and the actual outcome.
- It is calculated as the negative sum of the product of the true probability distribution and the
logarithm of the predicted probability distribution.
- Cross entropy is unbounded and can take values up to infinity. Lower values indicate better
model performance.

2. Brier Score:
- The Brier score measures the mean squared difference between the predicted probabilities
and the actual outcomes for each instance in the dataset.
- It is calculated as the average of the squared differences between the predicted probabilities
and the actual binary outcomes.
- The Brier score is bounded between 0 and 1, with 0 indicating perfect predictions and 1
indicating the worst possible predictions.
- Lower Brier scores indicate better calibration and accuracy of probabilistic predictions.

In summary, cross entropy and Brier score are both used to assess the quality of probabilistic
predictions, with cross entropy measuring the difference between probability distributions and
Brier score quantifying the mean squared difference between predicted probabilities and actual
outcomes. Both metrics are valuable for evaluating the calibration and accuracy of probabilistic
forecasts in classification tasks.

Topic 13: Performance evaluation – datasets

A three-way data split refers to the division of a dataset into three distinct subsets: training set,
validation set, and test set. Each subset serves a specific purpose in machine learning model
development and evaluation. Briefly, the three-way data split is as follows:

1. **Training Set:**
- Purpose: Used to train the machine learning model.
- Size: The largest portion of the dataset (typically around 70-80%).
- The model learns patterns, relationships, and features from this subset.

2. **Validation Set:**
- Purpose: Employed during the model development phase to fine-tune hyperparameters and
avoid overfitting.
- Size: Smaller than the training set (typically around 10-15%).
- The model's performance on the validation set helps guide adjustments to enhance
generalization.
3. **Test Set:**
- Purpose: Reserved for the final evaluation of the trained model's performance.
- Size: A portion independent of both training and validation sets (typically around 10-15%).
- The model's performance on the test set provides an unbiased assessment of its ability to
generalize to new, unseen data.

This three-way split helps ensure that the model is trained effectively, fine-tuned without
overfitting, and evaluated on a separate dataset to gauge its real-world performance.

The holdout method, also known as the holdout set or holdback validation, is a simple
technique in machine learning for model evaluation. In brief, it involves:

1. **Splitting the Dataset:**

- The original dataset is divided into two subsets: a training set and a holdout (validation or
test) set.

2. **Training the Model:**

- The machine learning model is trained exclusively on the training set.

3. **Evaluation on Holdout Set:**

- The performance of the trained model is then assessed on the holdout set, which was not
used during training.

4. **Model Adjustment:**
- Based on the performance on the holdout set, the model can be adjusted, hyperparameters
tuned, or further fine-tuned to improve generalization.

5. **Final Evaluation:**
- Once the model is fine-tuned, it is tested on an independent test set or deployed for real-
world predictions.

The holdout method helps prevent overfitting by providing an unbiased dataset for evaluating
model performance. It is a straightforward way to assess how well a model generalizes to new,
unseen data.

Stratification is a technique used in sampling and data splitting to ensure that each subgroup, or
stratum, within the population is represented proportionally in the sample or subsets. In brief:

1. **In Sampling:**
- When creating a sample from a population, stratification involves dividing the population
into distinct subgroups based on certain characteristics (strata), such as age, gender, or income.
- Samples are then drawn independently from each stratum, ensuring that each subgroup is
adequately represented in the final sample.
2. **In Data Splitting (e.g., for Cross-Validation):**
- In machine learning, when splitting a dataset into training and testing sets, stratification
ensures that the proportion of classes in each subset reflects the overall distribution in the
entire dataset.
- Particularly important in classification problems to avoid disproportionate representation of
classes in either the training or testing set.

Stratification helps improve the representativeness of samples or subsets, reducing the risk of
bias and ensuring that the characteristics of interest are adequately captured across different
strata.

K-fold cross-validation and random subsampling are techniques used to assess the performance
of machine learning models. Here's an overview of each:

1. K-fold Cross-Validation:
- K-fold cross-validation involves partitioning the dataset into k equally sized folds or subsets.
- The model is trained and evaluated k times, each time using a different fold as the validation
set and the remaining folds as the training set.
- The performance metrics from the k iterations are then averaged to obtain a robust estimate
of the model's performance.
- K-fold cross-validation is particularly useful for assessing how well a model generalizes to
new data and for obtaining a more reliable estimate of its performance compared to a single
train-test split.

2. Random Subsampling:
- Random subsampling, also known as simple holdout validation, involves randomly splitting
the dataset into a training set and a separate validation (or test) set.
- The model is trained on the training set and evaluated on the validation set to assess its
performance.
- This approach is simple and easy to implement, but it can lead to variability in the
performance estimate, especially when the dataset is small.
- Random subsampling is commonly used when the dataset is large enough to provide an
adequate representation of the underlying data distribution in both the training and validation
sets.

In summary, K-fold cross-validation involves partitioning the dataset into k subsets for repeated
model training and evaluation, while random subsampling entails a single random split of the
data into training and validation sets. Both techniques are valuable for assessing model
performance, with K-fold cross-validation providing a more robust estimate of performance
compared to random subsampling, especially when the dataset size is limited.
Leave-One-Out Cross-Validation (LOO-CV) is a model evaluation technique that involves using a
single observation from the original dataset as the validation data, and the remaining
observations as the training data. This process is repeated for each observation in the dataset,
and the performance of the model is averaged across all rounds of cross-validation.

LOO-CV is a special case of k-fold cross-validation where the number of folds is set to the
number of training instances. It has the advantage of making maximum use of the available data
and does not involve random subsampling. However, it can be computationally expensive,
especially for algorithms that are not easily updateable ,[object Object],.

One important point to note is that LOO-CV does not allow for stratification, meaning it does
not guarantee a stratified sample because there is only one instance in the test set. This can
lead to issues in cases where stratification is important for accurate model evaluation ,[object
Object],.

In summary, LOO-CV is a useful technique for model evaluation, especially when the dataset is
small, but it's important to be aware of its limitations, particularly in terms of computational
cost and the inability to perform stratification.

Bootstrapping is a resampling technique used in statistics and machine learning for estimating
the sampling distribution of a statistic by repeatedly sampling with replacement from the
observed data. In brief:

1. **Sampling with Replacement:**

- Randomly select data points from the observed dataset, allowing for the same data point to
be chosen more than once (with replacement).

2. **Creating Bootstrap Samples:**

- Generate multiple bootstrap samples by repeatedly drawing random samples from the
dataset.

3. **Statistical Estimation:**
- Calculate the desired statistic (e.g., mean, variance, confidence intervals) for each bootstrap
sample.

4. **Estimating Distribution:**
- Use the collection of computed statistics to estimate the sampling distribution of the statistic
of interest.

Bootstrapping is valuable when the underlying distribution of the data is unknown or complex.
It provides a non-parametric and empirical approach for making inferences about population
parameters or assessing the uncertainty associated with a statistical estimate.
The performance of datasets can be evaluated through various methods, depending on the
specific goals and characteristics of the data. Here are some common approaches to evaluating
dataset performance:

1. Descriptive Statistics: Analyzing the basic statistical properties of the dataset, such as mean,
median, variance, and distribution of the features, can provide insights into the overall
characteristics of the data.

2. Data Visualization: Creating visual representations of the data, such as histograms, scatter
plots, and box plots, can help in understanding the distribution and relationships between
variables within the dataset.

3. Outlier Detection: Identifying and analyzing outliers within the dataset can provide valuable
information about potential errors or anomalies in the data.

4. Data Quality Assessment: Evaluating the quality of the data in terms of completeness,
consistency, and accuracy can help in understanding the reliability of the dataset for analysis.

5. Feature Importance: Assessing the importance of different features within the dataset can
provide insights into which variables are most relevant for the analysis.

6. Cross-Validation: For predictive modeling tasks, using techniques such as k-fold cross-
validation can help in evaluating the performance of the dataset in training and testing the
model.

7. Model Evaluation: Assessing the performance of machine learning models trained on the
dataset can provide indirect insights into the quality and characteristics of the data.

It's important to consider the specific goals of the analysis and the nature of the dataset when
selecting the appropriate methods for evaluating dataset performance. Additionally, domain
knowledge and context should be taken into account to ensure a comprehensive evaluation ,[
],.

Considering costs in the context of machine learning involves taking into account the potential
costs associated with different types of prediction errors. This is particularly important in
scenarios where the costs of false positives and false negatives are not equal. Here are some
methods for considering costs in machine learning:

1. Cost-Sensitive Training: Most learning algorithms generate the same classifier regardless of
the cost matrix. However, it is possible to consider the cost matrix during training and ignore it
during prediction. Simple methods for cost-sensitive learning include resampling of instances
according to costs and weighting of instances according to costs. Additionally, some algorithms
can take costs into account by varying a parameter, such as naïve Bayes ,[object Object],.
2. Cost Matrix: The cost matrix is a key component in cost-sensitive learning. It defines the costs
associated with different types of prediction errors and is used to calculate the overall cost of
the predictor's performance. It is important to differentiate the cost matrix used for training
from the confusion matrix, which is used to evaluate the performance of a trained predictor
,[object Object],.

By incorporating cost considerations into the training process, machine learning models can be
optimized to minimize the overall cost of prediction errors, making them more effective in real-
world applications where different types of errors may have varying consequences.

**Internal Evaluation Measures for Clustering:**

1. **Silhouette Score:**
- Measures how similar an object is to its own cluster (cohesion) compared to other clusters
(separation).
- Ranges from -1 to 1, where higher values indicate better-defined clusters.

2. **Davies-Bouldin Index:**
- Computes the average similarity-to-dissimilarity ratio between clusters.
- Lower values indicate better clustering, with a minimum of 0.

3. **Inertia (or Within-Cluster Sum of Squares):**

- Measures the compactness of clusters by summing squared distances from each point to its
cluster center.
- Lower inertia suggests tighter, more cohesive clusters.

**External Evaluation Measures for Clustering:**

1. **Adjusted Rand Index (ARI):**
- Compares the similarity between true and predicted cluster assignments, adjusted for
chance.
- ARI ranges from -1 to 1, where 1 indicates perfect clustering.

2. **Normalized Mutual Information (NMI):**

- Measures the mutual information between true and predicted cluster assignments,
normalized by entropy.
- Ranges from 0 to 1, with higher values indicating better agreement.

3. **Fowlkes-Mallows Index:**
- Evaluates the similarity between true and predicted clusters using precision and recall.
- Combines precision and recall into a single measure, with 1 being a perfect match.

These metrics provide insights into different aspects of clustering quality, with internal
measures focusing on the inherent structure of the data and external measures assessing the
agreement between predicted clusters and ground truth.
MDL : The Minimum Description Length (MDL) principle is a concept in information theory and
statistics that provides a criterion for selecting a model or hypothesis among a set of competing
models. The MDL principle suggests that the best model is the one that minimizes the total
length of the description needed to represent both the model itself and the data given that
model.

In other words, the MDL principle aims to strike a balance between the complexity of a model
and its ability to explain the data. The idea is to penalize overly complex models that might
overfit the data by requiring longer descriptions, while favoring simpler models that can still
accurately represent the data with shorter descriptions. This principle is used in model
selection, data compression, and algorithmic information theory.