Artificial Intelligence

LEARNING: REGRESSION VS CLASSIFICATION

Week-14 Lecture-28
 Agenda 2

❑ Concept of Learning

❑ Regression

► Multiple Linear Regression.

► Polynomial Regression.

► Support Vector Regression.

❑ Classification

► Logistic Regression.

► k-nearest neighbors.
Learning
 4
Concept of Learning

► Self learning: Ability to recognize patterns, learn from data, and become more intelligent
over time (can be AI or programmatically based).
► Machine learning: It is an application of artificial intelligence (AI) that provides
systems the ability to automatically learn and improve from experience without being
explicitly programmed. Machine learning focuses on the development of computer
programs that can access data and use it to learn for themselves.
Regression
 6
Recap: Linear Regression

Dependent Variable Independent Variable

�
�
�� Constant Coefficient
Dependent
Variable

�
Independent Variable
�
��
 7
Example: Linear Regression

Salary Data
Example: Linear Regression (Ordinary Least 8
Squares)
 9
Multiple features (variables)

Size (feet2) Price ($1000)

2104 460
1416 232
1534 315
852 178
… …
 10
Multiple features (variables)

Size (feet2) Number of Number of Age of home Price ($1000)

bedrooms floors (years)

2104 5 1 45 460
1416 3 2 40 232
1534 3 2 30 315
852 2 1 36 178
… … … … …
 11
Multiple features (variables)

Size (feet2) Number of Number of Age of home Price ($1000)

bedrooms floors (years)

2104 5 1 45 460
1416 3 2 40 232
1534 3 2 30 315
852 2 1 36 178
… … … … …
Notation:
= number of features
= input (features) of training example.
= value of feature in training example.
 12
Hypothesis:

Previously:
(No More)

Now:
 13

For convenience of notation, define .

Multivariate linear regression.

 14
Housing prices prediction
 15
Polynomial regression

Price
(y)

Size
(x)
 16
Other Type of Regression

Support Vector Regression

Decision Tree Regression

Random Forest Regression

Artificial neural network regression

Deep learning based regression

 17
Example: Linear Regression (Implementation)

Importing Libraries

Data Import

Train and Test Split

 18
Example: Simple Linear Regression (Implementation)

Fitting a linear regressor

to the Salary Data

Plotting the Results

 19
Example: Simple Linear Regression (Results)
Classification
 21
Classification

❑ A classification problem is when the output variable is a category

“red” or “blue” or “disease” and “no disease”.
❑ A classification model attempts to draw some conclusion from
observed values.
❑ Given one or more inputs a classification model will try to predict
the value of one or more outcomes.

Which of the following is/are classification problem(s)?

▪ Predicting the gender of a person by his/her handwriting style

▪ Predicting house price based on area
▪ Predicting whether monsoon will be normal next year
▪ Predict the number of copies a music album will be sold next
Image source: Kaggle
month
Logistic Regression
 23
Logistic Regression: Overview

❑ Defined: A model for predicting one variable from

other variable(s).

❑ Variables: IV(s) is continuous/categorical,

DV is dichotomous (contrast b/w things)

❑ Relationship: Prediction of group membership

❑ Example: Can we predict students passage from GPA, etc.

❑ Assumptions: Multicollinearity (not linearity or normality)

 24
Comparison to Linear Regression

❑ Since dichotomous outcome, can’t use linear

regression because not linear

❑ Since dichotomous outcome, we are now talking

about “probabilities” (of 0 or 1)

❑ So logistic is about predicting the probability of

the outcome occurring.
 25
Logistic Regression basics

• Logistic is based upon “odds ratio”

• which is the probability of an event divided by probability of non-event.
• For example, if Exp(b) =2, then a one unit change would make the event
twice as likely (.67/.33) to occur.
 26
Logistic Regression basics

❑ Single predictor

❑ Multiple predictor

❑ Notice the linear regression equation

❑ e is the base of the natural logarithm (about 2.718)
 k-nearest neighbor: Intuition
27
❑ K nearest neighbors is a simple algorithm that stores all available
cases and classifies new cases based on a similarity measure (e.g.,
distance functions).

❑ KNN has been used in statistical estimation and pattern recognition

already in the beginning of 1970’s as a non-parametric technique.

Hasan, M.J.; Kim, J.-M. Fault Detection of a Spherical Tank Using a Genetic Algorithm-Based Hybrid Feature Pool and k-Nearest Neighbor Algorithm. Energies 2019, 12, 991.
Tree Based
Methods 28

Decision Tree Complexity

* Slides from Seo Hui(LG Electronics), “Gradient Boosting

Model”
 29
Implementation: Logistic Regression
 30
Implementation: Logistic Regression Cont.
 31
Implementation: Logistic Regression Cont.
 32
Logistic Regression: Training and Testing Results
 33
Another Classification Algorithms

Support vector machine

Random Forest
Naïve Bayes

Neural Network (MLP)

…
 34
References

► Y. Bengio, I. Goodfellow, and A. Courville, Deep learning, vol.1. Citeseer,

2017.ai.berkeley.edu

► SuperDataScience