Professional Documents
Culture Documents
BS Classes V2
BS Classes V2
The Model
yi = β0 + β1 xi + i , i = 1, 2, · · · , n (1)
3 / 69
Simple Regression Model Multiple Linear Regression Model
The Model
Remark
The word regression is curious, it means going backwards. In this
context we can interpret it as “returning to origins”, in the sense
that x is called a regressor because it somehow gives origin to y .
4 / 69
Simple Regression Model Multiple Linear Regression Model
5 / 69
Simple Regression Model Multiple Linear Regression Model
6 / 69
Simple Regression Model Multiple Linear Regression Model
Parameters Estimation
7 / 69
Simple Regression Model Multiple Linear Regression Model
Then the least squares method is to find βb0 and βb1 which
minimize the overall sum of squared differences between
observed and predicted values. So, taking partial derivatives,
equaling zero and solving:
nx y − ni=1 xi yi
P
βb1 = and βb0 = y − x βb1
nx 2 − ni=1 xi2
P
8 / 69
Simple Regression Model Multiple Linear Regression Model
Thus, βb0 and βb1 obtained by the least squares method are
unbiased estimators of β0 and β1 respectively.
Exercise 1
Calculate E[βb0 ], E[βb1 ], and if you’re brave, the variances.
9 / 69
Simple Regression Model Multiple Linear Regression Model
11 / 69
Simple Regression Model Multiple Linear Regression Model
βb1
t = qP ∼ tn−2,δ
n
s/ i=1 (xi − x )2
13 / 69
Simple Regression Model Multiple Linear Regression Model
14 / 69
Simple Regression Model Multiple Linear Regression Model
Coefficient of Determination r 2
Then
n
X n
X n
X
(yi − y )2 = (ybi − y )2 + (yi − ybi )2 (2)
i=1 i=1 i=1
15 / 69
Simple Regression Model Multiple Linear Regression Model
Coefficient of Determination r 2
Coefficient of Determination r 2
17 / 69
Simple Regression Model Multiple Linear Regression Model
MLRM: Formulation
y = Xβ + (3)
Where
y1 1 x11 x12 · · · x1k β0 1
y 1 x21 x22 · · · x2k β 2
2 1
y =
.. , X = .
. .. .. .. .. , β = . , = .
. .
. . . . . .
. .
yn 1 xn1 xn2 · · · xnk βk n
h i
Of course, X could also be written as X = j x 1 x2 · · · xk ,
h i0 h i0
where j = 1 1 · · · 1 and xi = x1i x2i ··· xni for
i = 1, 2, · · · , k.
18 / 69
Simple Regression Model Multiple Linear Regression Model
MLRM: Formulation
MLRM: Formulation
20 / 69
Simple Regression Model Multiple Linear Regression Model
MLRM: Formulation
21 / 69
Simple Regression Model Multiple Linear Regression Model
βb = (X 0 X)−1 X 0 y and
E[β]
b = β
b = σ 2 (X 0 X)−1
V[β]
24 / 69
Simple Regression Model Multiple Linear Regression Model
Normal model
25 / 69
Simple Regression Model Multiple Linear Regression Model
Normal model
Y
X
Figure 1: N2 (µ = 0, σ 2 = 1) 26 / 69
Simple Regression Model Multiple Linear Regression Model
Normal model
(y − Xβ)0 (y − Xβ)
l = log[L(β, σ 2 )] = − − log[(2πσ 2 )n/2 ]
2σ 2
Now we can find the maximum-likelihood estimators of the
parameters, deriving l with respect to β and σ 2 , equaling 0 and
clearing. So
βb = (X 0 X)−1 X 0 y
b 0 (y − X β)
(y − X β) b
b2 =
σ
n
27 / 69
Simple Regression Model Multiple Linear Regression Model
Normal model
1 βb ∼ N(β, σ 2 (X 0 X)−1 ).
2 b 2 /σ 2 ∼ χ2(n−k−1) , or equivalently
nσ
(n − k − 1)s 2 /σ 2 ∼ χ2(n−k−1) .
3 b 2 (or s 2 ) are independent.
βb and σ
28 / 69
Simple Regression Model Multiple Linear Regression Model
Sums of Squares
29 / 69
Simple Regression Model Multiple Linear Regression Model
Sums of Squares
i=1
n
yi = (1/n)j 0 y
X
y = (1/n)
i=1
ny 2 = n[(1/n)j 0 y]2 = n[(1/n)j 0 y][(1/n)j 0 y
but j 0 y = y 0 j so
ny 2 = (1/n)y 0 jj 0 y = (1/n)y 0 Jy
Then
n
(yi − y )2 = y 0 y − (1/n)y 0 Jy
X
i=1
= y 0 [I − (1/n)J]y
30 / 69
Simple Regression Model Multiple Linear Regression Model
Sums of Squares
Then:
SST = y 0 [I − (1/n)J]y
SSE = y 0 [I − H]y
SSR = y 0 [H − (1/n)J]y
Coefficient of Determination R 2
32 / 69
Simple Regression Model Multiple Linear Regression Model
Coefficient of Determination R 2
33 / 69
Simple Regression Model Multiple Linear Regression Model
34 / 69
Simple Regression Model Multiple Linear Regression Model
Q −1 y = Q −1 Xβ + Q −1
z = Wβ + δ (5)
35 / 69
Simple Regression Model Multiple Linear Regression Model
βb = (W 0 W )−1 W 0 z
b = σ 2 (W 0 W )−1
V[β]
z 0 [I − HW ]z
s2 = , with HW = W (W 0 W )−1 W 0
n−k −1
Exercise 4
Prove that
βb = (X 0 V −1 X)−1 X 0 V −1 y
b = σ 2 (X 0 V −1 X)−1
V[β]
y 0 [V −1 − V −1 X(X 0 V −1 X)−1 X 0 V −1 ]y
s2 =
n−k −1
36 / 69
Simple Regression Model Multiple Linear Regression Model
u/p
w = ∼ F(p,q)
v /q
u/p
w = ∼ F(p,q,λ)
v /q
37 / 69
Simple Regression Model Multiple Linear Regression Model
z
t = p ∼ t(p)
u/p
y
t = p ∼ t(p,µ)
u/p
y /σ
t = p ∼ t(p,µ/σ)
u/p
38 / 69
Simple Regression Model Multiple Linear Regression Model
39 / 69
Simple Regression Model Multiple Linear Regression Model
y = Xβ + subject to Cβ = 0
40 / 69
Simple Regression Model Multiple Linear Regression Model
βe = (X 0 X)−1 X 0 y
−(X 0 X)−1 C 0 [C(X 0 X)−1 C 0 ]−1 C(X 0 X)−1 X 0 y
And now
Now, do not forget that we are looking for a way to test the
null hypothesis H0 : Cβ = 0 and we must find a test statistic
that serves these purposes. With the theory we have
developed so far, it is logical to think about two independent
χ2 random variables. But note that:
y 0H ∗y = y 0 X(X 0 X)−1 C 0 [C(X 0 X)−1 C 0 ]−1 C(X 0 X)−1 X 0 y
b 0 [C(X 0 X)−1 C 0 ]−1 C β
= (C β) b
E[C β]
b = CE[β]
b = Cβ
V[C β] b 0 = σ 2 C(X 0 X)−1 C 0
b = CV[β]C
42 / 69
Simple Regression Model Multiple Linear Regression Model
43 / 69
Simple Regression Model Multiple Linear Regression Model
Finally
(y 0 H ∗ y/σ 2 )/q
F =
(y 0 (I − H)y/σ 2 )/(n − k − 1)
y 0 H ∗ y/q
= ∼ Fq,(n−k−1),λ
y 0 (I − H)y/(n − k − 1)
With λ = [1/(2σ 2 )](Cβ)0 [C(X 0 X)−1 C 0 ]−1 Cβ
44 / 69
Simple Regression Model Multiple Linear Regression Model
45 / 69
Simple Regression Model Multiple Linear Regression Model
(a 0 β)
b 0 [a 0 (X 0 X)−1 a]−1 (a 0 β)
b (a 0 β)
b 2
F = =
s2 s 2 [a 0 (X 0 X)−1 a]
∼ F1,(n−k−1),λ
(a 0 β)2
With λ =
2σ 2 a 0 (X 0 X)−1 a
Because q = 1, a 0 β,
b a 0 β and a 0 (X 0 X)−1 a are scalars. Then
we reject H0 if F ≥ Fα,1,(n−k−1) .
46 / 69
Simple Regression Model Multiple Linear Regression Model
βbj2 βj2
F = ∼ F 1,(n−k−1),λ with λ =
s 2 gjj 2σ 2 gjj
βbj
t = √ ∼ t(n−k−1),λ
s gjj
βbj − βj
t = √ ∼ t(n−k−1)
s gjj
P[|t| ≥ tα/2,(n−k−1) ] = α
P[|t| < tα/2,(n−k−1) ] = 1 − α
P[−tα/2,(n−k−1) < t < tα/2,(n−k−1) ] = 1 − α
48 / 69
Simple Regression Model Multiple Linear Regression Model
βbj − βj
P[−tα/2,(n−k−1) < √ < tα/2,(n−k−1) ]
s gjj
√ √
= P[−tα/2,(n−k−1) s gjj < βbj − βj < tα/2,(n−k−1) s gjj ]
√ √
= P[−βbj − tα/2,(n−k−1) s gjj < −βj < −βbj + tα/2,(n−k−1) s gjj ]
√ √
= P[βbj + tα/2,(n−k−1) s gjj > βj > βbj − tα/2,(n−k−1) s gjj ]
√ √
= P[βbj − tα/2,(n−k−1) s gjj < βj < βbj + tα/2,(n−k−1) s gjj ]
= 1−α
√
Then a 100(1 − α)% CI for βj is βbj ± tα/2,(n−k−1) s gjj .
49 / 69
Simple Regression Model Multiple Linear Regression Model
So:
" #
(n − k − 1)s 2 2 (n − k − 1)s 2
P ≤ σ ≤ = 1−α
χ2(α/2),(n−k−1) χ2(1−α/2),(n−k−1)
(n − k − 1)s 2 (n − k − 1)s 2
2 ≤ σ2 ≤ 2
χ(α/2),(n−k−1) χ(1−α/2),(n−k−1)
51 / 69
Simple Regression Model Multiple Linear Regression Model
h
HXi = X
h i
H j x1 · · · xk = j x1 · · · xk
⇒ Hj = j
53 / 69
Simple Regression Model Multiple Linear Regression Model
E[b] = 0 (6)
2
V[b] = σ (I − H) (7)
2
Cov[b, y] = σ (I − H) (8)
Cov[b, yb ] = 0 (9)
b0
b = j /n = 0 (10)
b0
y 0
= y (I − H)y (11)
b0 b
y = 0 (12)
b0
X = 0 0
(13)
54 / 69
Simple Regression Model Multiple Linear Regression Model
55 / 69
Simple Regression Model Multiple Linear Regression Model
due to (10) and (12). And due to (10) and (13) this
numerator is 0 for b and every column of the matrix X. Then
rb,by = 0
rb,xi = 0, i = 1, · · · , k
56 / 69
Simple Regression Model Multiple Linear Regression Model
Exercise 5
Using the Hematology Data in Table 10.1 of Rencher’s (p. 253),
described in the example 10.3, postule a “purely additive” linear
model for all explanatory variables and study the residuals.
57 / 69
Simple Regression Model Multiple Linear Regression Model
(1/n) ≤ hii ≤ 1
−0.5 ≤ hij ≤ 0.5, ∀i 6= j
X
tr(H) = hii = k + 1
58 / 69
Simple Regression Model Multiple Linear Regression Model
√
We need to scale the variance doing bi /[σ 1 − hii ] (the
standardized residuals) or the studentized residual
bi
ri = √
s 1 − hii
Our approach to checking for outliers is to plot studentized
residual versus ybi or versus i, the observation number.
There are different approaches to this analysis, however the
others are beyond the scope of this course.
60 / 69
Simple Regression Model Multiple Linear Regression Model
61 / 69
Simple Regression Model Multiple Linear Regression Model
60
40
y
20
20 30 40
x4
R code 1
# Scatter plot x4 vs y
ggplot(dat, aes(x = x4, y = y)) +
geom_point(colour=’darkblue’,shape=21,size=2)+
geom_smooth(method=’lm’,colour=’red’,se=TRUE)+
geom_point(aes(x=42,y=34),data=dat,size=10,
shape=1,color=’green’)+
geom_segment(aes(x=37,y=34,xend=41,yend=34),
lwd=0.2,col=’darkgreen’,arrow=arrow(length
=unit(0.03,"npc")))+
geom_point(aes(x=17, y=61),data=dat,size=10,
shape=1,color=’green’)+
geom_segment(aes(x=17,y=53,xend=17,yend=59),
lwd=0.2,col=’darkgreen’,arrow=arrow(length
=unit(0.03,"npc")))+
ylab(’y’)+
xlab(’x4’)
63 / 69
Simple Regression Model Multiple Linear Regression Model
64 / 69
Simple Regression Model Multiple Linear Regression Model
65 / 69
Simple Regression Model Multiple Linear Regression Model
b 0 X 0 X(β
(βb(i) − β) b − β)b
(i)
Di =
(k + 1)s 2
(X βb(i) − X β) b 0 (X β
b − X β)
b
(i)
=
(k + 1)s 2
(yb(i) − yb )0 (yb(i) − yb )
=
(k + 1)s 2
!
ri2 hii
=
k +1 1 − hii
66 / 69
Simple Regression Model Multiple Linear Regression Model
67 / 69
Simple Regression Model Multiple Linear Regression Model
R code 2
hv <- diag(H); hv
hatvalues(mod1)
lmax <- 2*(ncol(mod1$x)+1)/nrow(mod1$x); lmax
cd <- round(cooks.distance(mod1),6); cd
sx4 <- as.data.frame(sort(dat$x4, decreasing=T,
index.return=T))
sx4_y <- cbind(sx4, y=dat$y[sx4$ix], h=hv[sx4$ix],
c=cd[sx4$ix]); sx4_y
sy <- as.data.frame(sort(dat$y, decreasing=T,
index.return=T))
sy_x4 <- cbind(sy, x4=dat$x4[sy$ix], h=hv[sy$ix],
c=cd[sy$ix]); sy_x4
68 / 69
Simple Regression Model Multiple Linear Regression Model
69 / 69
Simple Regression Model Multiple Linear Regression Model
69 / 69