Regression Models

Dr. Amar Singh

Professor, School of Computer Applications
Lovely Professional University
Machine Learning :: Dr. Amar Singh
 Simple Linear Regression
• One Independent Variable is used to make a prediction
• Approximates Linear relationship between independent and dependent
variables.

One Independent Prediction Dependent

Variable X Variable Y

Experience Salary
4 20000
6 24000 If Experience = 5 Years then what
7 28000 would be the salary
9 32000
Machine Learning :: Dr. Amar Singh
2 14400
Linear Relationship between variables

Machine Learning :: Dr. Amar Singh

Simple Linear Regression
• The regression equation is a linear equation of the form:
• 𝟎 𝟏 . or ŷ = mx + c
• Where
• x is input variable value
• ŷ is predicted value
• 𝟎 : regression coefficient (c)
• 𝟏 : regression slope (m)

• In order to analyze ŷ we need to find 𝟎 and 𝟏

Fundamentals of Statistics
𝑪𝒐𝒗(𝑿,𝒀)
The correlation coefficient, r =
𝑽𝒂𝒓 𝑿 ∗𝑽𝒂𝒓(𝒀)

∑𝒏
𝒊 𝒙𝒊 𝒙
𝟐 ∑𝒏
𝒊 𝒙𝒊 𝒙
𝟐

𝒏 𝟏 𝒏 𝟏

∑𝒏
𝒊 𝒚𝒊 𝒚
𝟐

𝒏 𝟏
Covariance of X, Y
∑𝒏
𝒊 𝒙𝒊 𝒙 ∗ 𝒚𝒊 𝒚
=
𝒏 𝟏

In the simple Linear Regression Analysis with equation 𝟎 𝟏

𝟏 𝟎 𝟏
 Regression coefficient ( ), Regression slope ( )

• Regression coefficient ( 𝟏) is computed as follows :

𝒊 𝒊 𝒊
• 𝟏 𝟐
𝒊 𝒊

• are the mean values of X and Y.

• Regression slope ( 𝟎) can be calculated as :

•
Simple Linear Regression
• Last year, five randomly selected students
took a math aptitude test before they began Student xi yi
their statistics course. The Statistics 1 95 85
Department has three questions.
2 85 95
• What linear regression equation best predicts
statistics performance, based on math aptitude 3 80 70
scores? 4 70 65
• If a student made an 80 on the aptitude test,
5 60 70
what grade would we expect her to make in
statistics?
• How well does the regression equation fit the
data?
https://stattrek.com/regression/regression-example.aspx?Tutorial=reg
Calculating Intercept (b0) & Regression slope (b1)
Student xi yi 𝒊 𝒊 𝒊 𝒊 𝒊
𝟐

1 95 85 17 8 136 289
2 85 95 7 18 126 49
3 80 70 2 -7 -14 4
4 70 65 -8 -12 96 64
5 60 70 -18 -7 126 324
SUM 390 385 (Resuduals) (Resuduals) 470 730

MEAN 78 77

0.644
Regression equation is: ŷ = 0.644x + 26.768
 Predicting the output
• for x = 80 predicted value ŷ is computed as follows:
• ŷ = b0 + b1x
• ŷ = 26.768 + 0.644x
= 26.768 + 0.644 * 80
= 26.768 + 51.52
= 78.288

• Warning: When you use a regression equation, do not use values for the independent
variable that are outside the range of values used to create the equation. That is
called extrapolation, and it can produce unreasonable estimates.
• In this example, the aptitude test scores used to create the regression equation ranged
from 60 to 95. Therefore, only use values inside that range to estimate statistics grades.
Using values outside that range (less than 60 or greater than 95) is problematic.
Linear Regression
• Dependent Variable Y can be calculated as:
• Y = b0 + b1.X
Where is b0 intercept and b1 is Slope.

Machine Learning :: Dr. Amar Singh

Implementation of regression model in
python without sklearn

Machine Learning :: Dr. Amar Singh

Implementation of regression model in
python using sklearn

Machine Learning :: Dr. Amar Singh

Machine Learning :: Dr. Amar Singh
Plot Data

Machine Learning :: Dr. Amar Singh

Fit linear regression model

Machine Learning :: Dr. Amar Singh

What is the value of the intercept and slope

Machine Learning :: Dr. Amar Singh

Model Evaluation

Machine Learning :: Dr. Amar Singh

Plot Regression line

Machine Learning :: Dr. Amar Singh

Exercise
• Predict the weight of a person from his height.

Machine Learning :: Dr. Amar Singh

Multi-Regression Model
• Multiple predictors and one continuous valued target.

Machine Learning :: Dr. Amar Singh

Multiple Linear Regression
• = + 1 1+ 2 2+ 3 3+ 4 4
• :
• 1: 1
• 2: 2
• 3: 3
• 4: 4

Machine Learning :: Dr. Amar Singh

Multiple Linear Regression
• Consists of multiple regression coefficients (k independent variables
and k+1 regression coefficients).
X0 = 1
b0
X1 b1

b2 𝑌ℎ𝑎𝑡=b0+𝑏1𝑋1+𝑏2𝑋2+𝑏3𝑋3+ …….. + 𝑏k𝑋k

X2

b3
X3
…..
….. b4
X1

Machine Learning :: Dr. Amar Singh

To express the regression equation in matrix form, we need to define three
matrices: Y, b, and X.

the multiple regression equation can be

expressed concisely as:
Y = Xb
The least squares normal equations can
be expressed as:
X'Y = X'Xb or X'Xb = X’Y

To solve for regression coefficients,

simply pre-multiply by the inverse
of X'X:
(X'X)-1X'Xb = (X'X)-1X’Y

b = (X'X)-1X’Y
where (X'X)-1X'X = I, the identity matrix.

Machine Learning :: Dr. Amar Singh

Machine Learning :: Dr. Amar Singh
Machine Learning :: Dr. Amar Singh
Machine Learning :: Dr. Amar Singh
Implementation

Machine Learning :: Dr. Amar Singh

Non-Linear Regression

Machine Learning :: Dr. Amar Singh

Non-Linear Regression

Machine Learning :: Dr. Amar Singh

Machine Learning :: Dr. Amar Singh
Machine Learning :: Dr. Amar Singh
Machine Learning :: Dr. Amar Singh
Machine Learning :: Dr. Amar Singh