Probability and Statistics (MATH F113)

Introduction to Regression

Himadri Mukherjee

Department of Mathematics
BITS PILANI K K Birla Goa Campus, Goa

April 27, 2024

 Introduction

The regression curve Recall that in a joint distribution of random

variables X, Y , the curve µY |x is called the curve of regression.
 Linear regression model

Definition
There are parameters β0 , β1 , σ 2 , such that for any fixed value of
the independent variable x the dependent variable is a random
variable related to x through the model

Y = β0 + β1 x + 

The quantity  is a random variable assumed to be normally

distributed with E() = 0 and V () = σ 2 .

Note the random variable  is called the random deviation or

random error of the model.
Regression
 Model parameters

Let us define a few quantities

• Sxy =
P P P
xi yi − ( xi yi )/n
P 2
• Sxx = xi − ( xi )2 /n
P

• SST ”total sum of squares”=Syy = yi2 − ( yi )2 /n

P P

• SSE ”error sum of squares”= (yi − yˆi )2

P

•
(yi − yˆi )2
P
2 2 SSE
s = σ̂ = =
n−2 n−2
• Coefficient of determination r2 = 1 − SSE SST
P 2 ˆ
• SSE= yi − β0 yi − β1 xi yi
P ˆ P

• (yi − yˆi ) are called the residual values.

 The parameters

The fitted regression line is y = βˆ0 + βˆ1 x, where the estimators

are given by:
P
ˆ (xi − x)(yi − y) Sxy
β1 = P =
(xi − x) Sxx
yi − βˆ1 xi
P P
βˆ0 = = y − βˆ1 x
n
 A few words on σ̂ 2 and r2 etc

SSE follows a σ 2 χ2n−2 distribution. Equivalently, SSE

σ2
follows a
SSE
χn−2 distribution. S = n−2 is an unbiased estimator of σ 2 .
2 2
 Continued

The coefficient of determination r2 quantifies the amount of

observed variation of Y that can be attributed to a linear relation
between X, Y .
 Continued

If the r2 value is higher, the linear model is more suited, smaller

r2 means other models are needed or there are other variables
involved.
 Continued

Be careful about rounding errors, as small errors magnify

eventually because of the involvement of small numbers.
Example 1.
Continued