Linear Regression - 1

How does the data look for Linear Regression?

(𝑛 ×𝑑)
Regression Problem → Data: n samples with [𝑓1, 𝑓2,....., 𝑓𝑑]ϵ 𝑅 where
- sample 𝑋𝑖 consists the features
- That has a target label 𝑦𝑖 ϵ 𝑅 .

How does the training data look for the Regression problem?

Regression → Supervised task, the target 𝑦𝑖 is numerical

𝑛𝑋𝑑 𝑇
1. Input sample = 𝑋𝑖 ϵ 𝑅 , 𝑥𝑖 = [𝑥𝑖1,....., 𝑥𝑖𝑑]
2. Output sample = 𝑦𝑖 ϵ 𝑅
3. Training example = (𝑋𝑖,𝑦𝑖)
4. Training Dataset = {(𝑋𝑖,𝑦𝑖), 𝑖 = 1, 2,.. 𝑛} ,

What will be the simplest model for predicting a value?

Ans: Mean model → the mean of the entire data as its prediction.
What is the goal of the ML model?

Ans: To find 𝑓: 𝑋 → 𝑦 such that 𝑓(𝑥𝑖) ≈ 𝑦𝑖

How do we define function f?

^
Algebraic Intuition → find 𝑦 = 𝑓(𝑥𝑖),

we can say for Linear Regression:

𝑓(𝑥𝑖1, 𝑥𝑖2, ..... , 𝑥𝑖𝑑) = 𝑤1𝑥𝑖1 + 𝑤2𝑥𝑖2 + 𝑤3𝑥𝑖3 +.... + 𝑤𝑑𝑥𝑖𝑑 + 𝑤0
𝐷
^
𝑦𝑖 = 𝑓( 𝑥𝑖) = ∑ 𝑤𝑗𝑥𝑖𝑗 + 𝑤0
𝑗=1

𝑇
Now, 𝑤 = [𝑤1, 𝑤2..... 𝑤𝑑] and 𝑥𝑖 = [𝑥𝑖1, 𝑥𝑖2..... 𝑥𝑖𝑑] , then:
^ 𝑇
𝑦𝑖 = 𝑓(𝑥𝑖) = 𝑤 𝑥𝑖 + 𝑤0

How does the ML model find the function f?

Ans: By updating the weights of the model on the training dataset

Is 𝑓(𝑥𝑖) in Linear Regression analogous to 𝑦 = 𝑚𝑥 + 𝑐 ?

Ans: Yes, it is.

Linear Regression: finding the best D Dimensional hyperplane that fits the
^
D-dimensional data such that 𝑦𝑞 ≈ 𝑦𝑞

How to find the best-fit line of the Linear Regression model?

𝑇
Ans: By optimizing the weights vector 𝑊 = [𝑤1, 𝑤2,.....,, 𝑤𝑑] w.r.t the loss
function.
How do you say Linear Regression is optimized?

Ans: when we see the loss function is not decreasing anymore i.e. local minima

Sklearn-code

What loss function to use for linear regression optimization?

^
Mean Square Error→finds the mean of the square difference between 𝑦 , 𝑦 .
𝑛
1 ^ 2
𝑚𝑖𝑛𝑤,𝑤 𝑛
∑ (𝑦𝑖 − 𝑦𝑖)
0 𝑖=1

After training the model, how do we measure model performance?

Ans: R-squared metric. → measures the performance of Linear Regression

over a mean model. It is Defined as:
𝑛
^ 2
𝑆𝑆𝑟𝑒𝑠 ∑ (𝑦𝑖−𝑦𝑖)
2 −
𝑅 = 1 − 𝑆𝑆𝑡𝑜𝑡𝑎𝑙
= 1− 𝑖=1
𝑛 , where 𝑦 𝑖 is the mean model
− 2
∑ (𝑦𝑖−𝑦𝑖 )
𝑖=1

SSres - Squared sum of error of regression line

SStotal-(total sum of squares) Squared sum of error of mean line

Range: Practically, R2 ranges from (0, 1]

Sklearn code
 2
What will be the best value of 𝑅 ?
𝑛
^ 2
Ans: 1, when ∑ (𝑦𝑖 − 𝑦𝑖) = 0.
𝑖=1

2
What will be the minimum value of 𝑅 ?
𝑛 𝑛
^ 2 − 2
Ans: - ∞ , when ∑ (𝑦𝑖 − 𝑦𝑖) >> ∑ (𝑦𝑖 − 𝑦𝑖 )
𝑖=1 𝑖=1