L10a - Machine Learning Basic Concepts

Machine Learning: Basic Concepts Artificial Intelligence
Machine Learning:
Basic Concepts
by Samuel I. G. Situmeang
Modified slides provided by: Ansaf Salleb-Aouissi

Artificial Intelligence, Columbia University, 2018
Objectives
• Machine Learning Concepts

• Training-Testing
• Overfitting-Underfitting and Regularization
2
Machine Learning Concepts
3
Terminology
Machine Learning, Data Science, Data Mining, Data Analysis, Statistical

Learning, Knowledge Discovery in Databases, Pattern Discovery.
4
A Few Quotes
• "A breakthrough in machine learning would be worth ten Microsofts"

(Bill Gates, Chairman, Microsoft)
• "Machine learning is the next Internet"
(Tony Tether, former Director, DARPA - Defense Advanced Research Projects Agency)
• Machine learning is the hot new thing"
(John Hennessy, former President, Stanford)
• "Web rankings today are mostly a matter of machine learning"
(Prabhakar Raghavan, former Dir. Research, Yahoo)
• "Machine learning is going to result in a real revolution"
(Greg Papadopoulos, CTO, Sun Microsystems)
• "Machine learning is today’s discontinuity"
(Jerry Yang, former CEO, Yahoo)
5
Data everywhere!
Source: https://www.domo.com/
6
Data types
Data comes in different sizes and also flavors (types):

 Texts
 Numbers
 Clickstreams
 Graphs
 Tables
 Images
 Transactions
 Videos
 Some or all of the above!
7
Smile, we are 'DATAFIED'!
• Wherever we go, we are "datafied".

• Smartphones are tracking our locations.
• We leave a data trail in our web browsing.
• Interaction in social networks.
• Privacy is an important issue in Data Science.
8
The Data Science process
9
Applications of ML
• We all use it on a daily basis. Examples:
10
Applications of ML
• Spam filtering
• Credit card fraud detection
• Digit recognition on checks, zip codes
• Detecting faces in images
• MRI image analysis
• Recommendation system
• Search engines
• Handwriting recognition
• Scene classification
• etc...
11
Now is a great time to study ML
Research progress
Image classification Audio synthesis Games
Products
Voice recognition Translation Self-driving cars

12
Progress in ML is driven by…
• More compute
• More data
• Better algorithms → Need more people who understand the
algorithms!
13
Interdisciplinary field
14
ML versus Statistics
Statistics: Machine Learning:

• Hypothesis testing • Decision trees
• Experimental design • Rule induction
• Analysis of variance (ANOVA) • Neural Networks
• Linear regression • Support Vector Machines (SVMs)
• Logistic regression • Clustering method
• Generalized Linear Models (GLM) • Association rules
• Principal Component Analysis (PCA) • Feature selection
• Visualization
• Graphical models
• Genetic algorithm
http://statweb.stanford.edu/~jhf/ftp/dm-stat.pdf
15
Machine Learning definition
Alan Turing proposed the concept of a learning machine in 1950 (in the
same paper that proposed the Turing test).
Idea: Divide the problem into two parts:

1. A machine that simulates a child’s brain (analogous to a blank notebook:
should function by simple mechanisms and have lots of blank sheets).
2. A way of teaching the child machine (should be simple since we know how
to teach a human child).
Teacher rewards good behaviour and penalizes bad behaviour.
16
"An important feature of a learning machine is that its teacher will

often be very largely ignorant of quite what is going on inside."
Alan Turing
• While we don’t know how our brain converts input to output, we

know what the output should be for every input.
• We can use this knowledge to teach the machine.
17
“How do we create computer programs that improve with

experience?”
Tom Mitchell
http://videolectures.net/mlas06_mitchell_itm/
18
"How do we create computer programs that improve with

experience?"
Tom Mitchell
http://videolectures.net/mlas06_mitchell_itm/
"A computer program is said to learn from experience 𝐸 with respect

to some class of tasks 𝑇 and performance measure 𝑃, if its
performance at tasks in 𝑇, as measured by 𝑃, improves with
experience 𝐸."
Tom Mitchell. Machine Learning 1997.
19
• A branch of artificial intelligence, concerned with the design and

development of algorithms that allow computers to evolve behaviors
based on empirical data.
• As intelligence requires knowledge, it is necessary for computers to

acquire knowledge.
Data
• Traditional programming Computer Output
Program
• Machine learning Data

Computer Program
Output
20
Types of machine learning Algorithms
There some variations of how to define the types of Machine Learning

Algorithms but commonly they can be divided into categories
according to their purpose and the main categories are the following:
• Supervised learning (predictive model, "labeled" data).
• Classification
• Numeric prediction/forecasting/regression
• Unsupervised learning (descriptive model, "unlabeled" data).
• Clustering
• Pattern Discovery
• Semi-supervised learning (mixture of "labeled" and "unlabeled" data).
• Reinforcement learning. Using this algorithm, the machine is trained to make
specific decisions. It works this way: the machine is exposed to an
environment where it trains itself continually using trial and error. This
machine learns from past experience and tries to capture the best possible
knowledge to make accurate business decisions.
21
Common algorithms for each categories
• Supervised learning
• Classification. e.g. Logistic Regression, Decision Tree, KNN, Random Forest,
SVM, & Naive Bayes
• Numeric prediction/forecasting/regression. e.g. Linear Regression, KNN,
Gradient Boosting & AdaBoost
• Unsupervised learning
• Clustering. e.g. K-Means
• Pattern Discovery. e.g. Apriori, FP-Growth, & Eclat
• Semi-supervised learning
• Reinforcement learning.
• e.g. Q-Learning, Temporal Difference (TD), & Deep Adversarial Networks
22
A brief of ML types with sample use cases
Source: https://en.proft.me/
23
Supervised vs. Unsupervised
Given: Training data: 𝑥1 , 𝑦1 , … , 𝑥𝑛 , 𝑦𝑛 Τ𝑥𝑖 ∈ ℝ𝑑 and 𝑦𝑖 is the label.

example 𝑥1 → 𝑥11 𝑥12 … 𝑥1𝑑 𝑦1 ← label
… … … … … …
example 𝑥𝑖 → 𝑥𝑖1 𝑥𝑖2 … 𝑥𝑖𝑑 𝑦𝑖 ← label
… … … … … …
example 𝑥𝑛 → 𝑥𝑛1 𝑥𝑛2 … 𝑥𝑛𝑑 𝑦𝑛 ← label
fruit length width weight label

fruit 1 165 38 172 Banana
fruit 3 76 80 145 Orange
…
fruit n … … … …
24
Supervised vs. Unsupervised
fruit length width weight label

…
fruit n … … … …
Unsupervised learning:
Learning a model from unlabeled data.
Supervised learning:
Learning a model from labeled data.
25
Unsupervised Learning
Training data: "examples" 𝑥.

𝑥1 , … , 𝑥𝑛 , 𝑥𝑖 ∈ 𝑋 ⊂ ℝ𝑛
• Clustering/segmentation:
𝑓: ℝ𝑑 → 𝐶1 , … , 𝐶𝑘 set of cluster
Example: Find clusters in the population, fruits, species.
26
Clustering/segmentation:
27
28
Methods: K-means, gaussian mixtures, hierarchical clustering, spectral

clustering etc.
29
Supervised Learning
Training data: "examples" 𝑥 with "labels" 𝑦.

𝑥1 , 𝑦1 , … , 𝑥𝑛 , 𝑦𝑛 Τ𝑥𝑖 ∈ ℝ𝑑
• Classification: 𝑦 is discrete. To simplify, 𝑦 ∈ −1. +1
𝑓: ℝ𝑑 → −1, +1 𝑓 is called a binary classifier
Example: Approve credit yes/no, spam/ham, banana/orange.
30
Supervised Learning
Classification:
31
Supervised Learning
Classification:
32
Supervised Learning
Classification:
Methods: Support Vector Machines, neural networks, decision trees,

K-nearest neighbors, naive Bayes, etc.
33
Supervised Learning
Classification:
34
Supervised Learning
Non linear classification
35
Supervised Learning
Training data: "examples" 𝑥 with "labels" 𝑦.

𝑥1 , 𝑦1 , … , 𝑥𝑛 , 𝑦𝑛 Τ𝑥𝑖 ∈ ℝ𝑑
• Regression: 𝑦 is a real value, 𝑦 ∈ ℝ
𝑓: ℝ𝑑 → ℝ 𝑓 is called a regressor
Example: amount of credit, weight of fruit.
36
Supervised Learning
Regression:
Example: Income in function of age, weight of the fruit in function

of its length.
37
Supervised Learning
Regression:
38
Supervised Learning
Regression:
39
Supervised Learning
Regression:
40
Training and Testing
41
Training set
ML Algorithm
Model (f)
42
Training set
ML Algorithm
Income,
gender,
age, Credit amount $
family Model (f) Credit yes/np
status,
zipcode
43
Training set
ML Algorithm
Income,
gender,
age, Credit amount $
family Model (f) Credit yes/np
status,
zipcode
Question: How can we be confident about 𝒇?
44
• We calculate 𝐸 𝑡𝑟𝑎𝑖𝑛 the in-sample error (training error or empirical

error/risk). prediction
𝑛
label
𝑡𝑟𝑎𝑖𝑛
𝐸 𝑓 = ෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖
𝑖=1 true label
45

error/risk).
𝑛
𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 = ෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖
𝑖=1
• Examples of loss functions:
• Classification error:
1 𝑠𝑖𝑔𝑛 𝑦𝑖 ≠ 𝑠𝑖𝑔𝑛 𝑓 𝑥𝑖
𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 =ቊ
0 otherwise
46

error/risk).
𝑛
𝑖=1
• Examples of loss functions:
• Classification error:
1 𝑠𝑖𝑔𝑛 𝑦𝑖 ≠ 𝑠𝑖𝑔𝑛 𝑓 𝑥𝑖
𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 =ቊ
0 otherwise
• Least square loss:
2
𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 = 𝑦𝑖 − 𝑓 𝑥𝑖
47

error/risk).
𝑛
𝑖=1
• We aim to have 𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 small, i.e., minimize 𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓
• We hope that 𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 , the out-sample error (test/true error), will

be small too.
48
Overfitting-Underfitting and
Regularization
49
Overfitting/underfitting
• Overfitting: keadaan ketika model memiliki kinerja baik hanya untuk

training data/seen examples tetapi tidak memiliki kinerja baik untuk
unseen examples.
• Terjadi ketika model terlalu fleksibel (memiliki kemampuan yang terlalu tinggi
untuk mengestimasi banyak fungsi) atau terlalu mencocokkan diri terhadap
training data.
• Underfitting: keadaan ketika model memiliki kinerja buruk baik untuk
training data dan unseen examples.
• Terjadi akibat model yang telalu tidak fleksibel (memiliki kemampuan yang
rendah untuk mengestimasi variasi fungsi.
50
Structural Risk Minimization

High Bias Low Bias
Prediction Error LowVariance High Variance
Test error
Training error
UnderﬁAng Good models OverﬁAng
Low Complexity of the model High
51
52
53
Avoid overfitting
• In general, use simple models!

• Reduce the number of features manually or do feature selection.
• Do a model selection (ML course).
• Use regularization (keep the features but reduce their importance by setting
small parameter values) (ML course).
• Do a cross-validation to estimate the test error.
54
Regularization: Intuition
We want to minimize:
Classification term + 𝐶 × Regularization term
෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 +𝐶×𝑅 𝑓
𝑖=1
55
Regularization: Intuition
Hint: Avoid high-degree polynomials.
56
Train, Validation and Test
Example: Split the data randomly into 60% for training, 20% for
validation and 20% for testing.
Source: https://towardsdatascience.com/
57
Training set is a set of examples used for learning a model (e.g., a

classification model).
58

Validation set is a set of examples that cannot be used for learning the
model but can help tune model parameters (e.g., selecting K in K-NN).
Validation helps control overfitting.
59

Test set is used to assess the performance of the final model and
provide an estimation of the test error.
60

Test set is used to assess the performance of the final model and
provide an estimation of the test error.
Note: Never use the test set in any

way to further tune the parameters
or revise the model.
61
Classifier Evaluation Method: Holdout &

Cross-Validation
• Holdout method
• Given data is randomly partitioned into two independent sets
• Training set (e.g., 2/3) for model construction
• Test set (e.g., 1/3) for accuracy estimation
• Repeated random sub-sampling validation: a variation of holdout
• Repeat holdout 𝑘 times, accuracy = avg. of the accuracies
obtained
• Cross-validation (𝑘-fold, where 𝑘 = 10 is most popular)
• Randomly partition the data into 𝑘 mutually exclusive subsets, each
approximately equal size
• At 𝑖-th iteration, use 𝐷𝑖 as test set and others as training set
• Leave-one-out: 𝑘 folds where 𝑘 = # of tuples, for small sized data
• *Stratified cross-validation*: folds are stratified so that class
distribution, in each fold is approximately the same as that in the
initial data
62
K-fold Cross Validation
A method for estimating test error using training data.

Algorithm:
Given a learning algorithm 𝒜 and a dataset 𝒟
Step 1: Randomly partition D into k equal-size subsets D1; : : : ; Dk
Step 2:
For 𝑗 = 1 to 𝑘
Train 𝒜 on all 𝐷𝑖 , 𝑖 ∈ 1, … , 𝑘 and 𝑖 ≠ 𝑗, and get 𝑓𝑗
Apply 𝑓𝑗 to 𝐷𝑗 and compute 𝐸𝐷𝑗
Step 3: Average error over all folds.
𝑘
෍ 𝐸 𝐷𝑗
𝑗=1
63
Confusion matrix
64
Classification Model Evaluation Metrics
𝐅𝟏 − 𝐒𝐜𝐨𝐫𝐞 2𝑃𝑅 Harmonic mean of precision and recall

𝑃+𝑅
65
Evaluation: Binary Classification
• Accuracy:
• How many of the samples are classified correctly? C1
• A = 9/10 = 0.9
C2
Thursday, Apr. 12, 2018 SGS/ ISS4011-Sistem Temu Balik Informasi 66


System 2 System 1
• A = 7/10 = 0.7 System 1
System 2 C1 C1
• A = 7/10 = 0.7
• When classes are highly unbalanced
• Precision/recall/F1 for the rare class
• e.g. Spam classification (detection)
C2 C2


System 1 System 2 System 2 System 1
Precision 1/3 = 0.33 0/1 = 0 C1 C1
Recall 1/2 = 0.5 0/2 = 0
F1 0.4 0
C2 C2

Error Analysis
𝑪𝟏
• Confusion Matrix
• How classes get confused?
Predicted
𝐶1 𝐶2 𝐶3 𝑪𝟐
3 0 1
Actual 0 3 1
1 0 1 𝑪𝟑
• Useful:
• Find classes that get confused with others
• Develop better features to solve the problem

Evaluation: Multi-class
𝑪𝟏
𝐶1 𝐶2 𝐶3
P 0.75 1 0.333
R 0.75 0.75 0.5
F1 0.75 0.86 0.4 𝑪𝟐
• Accuracy = (3+3+1)/10 = 0.7

• Good measure when 𝑪𝟑
• Classes are nearly balanced
• Preferred:
• Precision/recall/F1 for each class

σ𝑁
𝑖=1 TP𝑖
• Micro-average of Precision= σ𝑁 𝑁
𝑖=1 TP𝑖 +σ𝑖=1 FP𝑖
σ𝑁
𝑖=1 TP𝑖
• Micro-average of Recall= σ𝑁 𝑁
𝑖=1 TP𝑖 +σ𝑖=1 FN𝑖
σ𝑁
𝑖=1 P𝑖
• Macro-average of Precision=
𝑁
σ𝑁
𝑖=1 R𝑖
• Macro-average of Recall=
𝑁
71
𝑪𝟏
• Majority class baseline
• Accuracy = 0.8
• Macro-F1 = 0.296
𝑪𝟐
• Macro-F1:
• Should be used in binary classification when
two classes are important
• e.g.: males/females while distribution is 80/20% 𝑪𝟑

Example of Confusion Matrix:
Use the same confusion matrix, calculate the measure just introduced.
Actual Class\Predicted class cancer = yes cancer = no Total Recognition(%)
cancer = yes 90 210 300 30.00 (sensitivity)
cancer = no 140 9560 9700 98.56 (specificity)
Total 230 9770 10000 96.50 (accuracy)
• Sensitivity = TP/P = 90/300 = 30%

• Specificity = TN/N = 9560/9700 = 98.56%
• Accuracy = (TP + TN)/All = (90+9560)/10000 = 96.50%
• Error rate = (FP + FN)/All = (140 + 210)/10000 = 3.50%
• Precision = TP/(TP + FP) = 90/(90 + 140) = 90/230 = 39.13%
• Recall = TP/ (TP + FN) = 90/(90 + 210) = 90/300 = 30.00%
73
Regression Model Evaluation Metrics
• Mean Squared Error (MSE)

• Root Mean Squared Error (RMSE)
• Mean Absolute Error (MAE)
• R Squared (R²)
• Adjusted R Squared (R²)
• Mean Square Percentage Error (MSPE)
• Mean Absolute Percentage Error (MAPE)
• Root Mean Squared Logarithmic Error (RMSLE)
74
Mean Squared Error (MSE)
• It is perhaps the most simple and common metric for regression

evaluation, but also probably the least useful. It is defined by the
equation.
• where 𝑦𝑖 is the actual expected output and 𝑦ො𝑖 is the model’s prediction.
• The higher this value, the worse the model is. It is never negative,
since we’re squaring the individual prediction-wise errors before
summing them, but would be zero for a perfect model.
75
Root Mean Squared Error (RMSE)
• RMSE is the square root of MSE. The square root is introduced to

make scale of the errors to be the same as the scale of targets.
𝑁
1 2
RMSE = ෍ 𝑦𝑖 − 𝑦ො𝑖 = MSE
𝑁
𝑖=1
76
Terminology review
Review the concepts and terminology:
Instance, example, feature, label, supervised learning, unsupervised

learning, classification, regression, clustering, prediction, training set,
validation set, test set, K-fold cross validation, classification error, loss
function, overfitting, underfitting, regularization.
77
Machine Learning Books
1. Tom Mitchell, Machine Learning.

2. Abu-Mostafa, Yaser S. and Magdon-Ismail, Malik and Lin, Hsuan-
Tien, Learning From Data, AMLBook.
3. The elements of statistical learning. Data mining, Inference, and
Prediction T. Hastie, R. Tibshirani, J. Friedman.
4. Christopher Bishop. Pattern Recognition and Machine Learning.
5. Richard O. Duda, Peter E. Hart, David G. Stork. Pattern
Classification. Wiley
78
Machine Learning Resources
• Major journals/conferences: ICML, NIPS, UAI, ECML/PKDD, JMLR,

MLJ, etc.
• Machine learning video lectures:
http://videolectures.net/Top/Computer_Science/Machine_Learning/
• Machine Learning (Theory):
http://hunch.net/
• LinkedIn ML groups: \Big Data" Scientist, etc.
• Women in Machine Learning:
https://groups.google.com/forum/#!forum/women-in-machine-learning
• KDD nuggets
http://www.kdnuggets.com/
79
References
• T. Mitchell, Machine Learning, 1997.

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical
Learning: Data Mining, Inference, and Prediction (2nd Edition), 2009.
• S. J. Russell and P. Borvig, Artificial Intelligence: A Modern Approach
(3rd Edition), Prentice Hall International, 2010.
80

L10a - Machine Learning Basic Concepts

Uploaded by

Copyright:

Available Formats

You might also like

L10a - Machine Learning Basic Concepts

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

L10a - Machine Learning Basic Concepts

Uploaded by

Copyright:

Available Formats

Machine Learning: Basic Concepts Artificial Intelligence

Modified slides provided by: Ansaf Salleb-Aouissi

• Machine Learning Concepts

Machine Learning Concepts

Machine Learning, Data Science, Data Mining, Data Analysis, Statistical

• "A breakthrough in machine learning would be worth ten Microsofts"

Data comes in different sizes and also flavors (types):

Smile, we are 'DATAFIED'!

• Wherever we go, we are "datafied".

The Data Science process

• We all use it on a daily basis. Examples:

Now is a great time to study ML

Image classification Audio synthesis Games

Voice recognition Translation Self-driving cars

Progress in ML is driven by…

Statistics: Machine Learning:

Machine Learning definition

Idea: Divide the problem into two parts:

Teacher rewards good behaviour and penalizes bad behaviour.

Machine Learning definition

"An important feature of a learning machine is that its teacher will

• While we don’t know how our brain converts input to output, we

Machine Learning definition

“How do we create computer programs that improve with

Machine Learning definition

"How do we create computer programs that improve with

"A computer program is said to learn from experience 𝐸 with respect

Machine Learning definition

• A branch of artificial intelligence, concerned with the design and

• As intelligence requires knowledge, it is necessary for computers to

• Machine learning Data

Types of machine learning Algorithms

There some variations of how to define the types of Machine Learning

Common algorithms for each categories

A brief of ML types with sample use cases

Supervised vs. Unsupervised

Given: Training data: 𝑥1 , 𝑦1 , … , 𝑥𝑛 , 𝑦𝑛 Τ𝑥𝑖 ∈ ℝ𝑑 and 𝑦𝑖 is the label.

fruit length width weight label

Supervised vs. Unsupervised

fruit length width weight label

Training data: "examples" 𝑥.

Example: Find clusters in the population, fruits, species.

Methods: K-means, gaussian mixtures, hierarchical clustering, spectral

Training data: "examples" 𝑥 with "labels" 𝑦.

• Classification: 𝑦 is discrete. To simplify, 𝑦 ∈ −1. +1

𝑓: ℝ𝑑 → −1, +1 𝑓 is called a binary classifier

Example: Approve credit yes/no, spam/ham, banana/orange.

Methods: Support Vector Machines, neural networks, decision trees,

Non linear classification

Training data: "examples" 𝑥 with "labels" 𝑦.

• Regression: 𝑦 is a real value, 𝑦 ∈ ℝ

Example: amount of credit, weight of fruit.

Example: Income in function of age, weight of the fruit in function

Training and Testing

Training and Testing

Training and Testing

Training and Testing