STA302/1001 - Methods of Data Analysis I

(Week 08 lecture notes)

Wei (Becky) Lin

Oct. 31, 2016

1/50
Midterm and Final

• If you wrote the midterm, you should receive a mail from Crowdmark.
• For remark request:
• Write me an email within 7 days.
• The request must contain a justification for consideration.
• All remark requests, the whole assignment/ test will be
remarked. There is chance that you might get lower mark points
after remarking.
• Final is on Dec. 12, AM 9-12
• A to J: BN 2N
• K to R: BN 2S
• S to ZHE: BN 3
• Zho to ZZ: ZZ VLAd

2/50
Last Week

• Variable transformations.
• More on logarithmic transformation.
• Box-Cox transformation.
• Interpretation of slope after transformation.
• Chapter 4: Simultaneous Inferences

3/50
Review: from p-value to conclusion
True model Estimated model

Yi = —0 + —1 Xi + ‘i ; Ŷi = b0 + b1 Xi
H0 : —1 = 0; Ha : —1 ”= 0

• test statistic:
b1
tú = |H ≥ tn≠2 b. s×¥× scb.t.tn#xps
s(b1 ) 0
=

• Let the P-value of this test be Pt , assume significance level, – = 5%

• Case 1: Pt < 0.05
• We have weak/moderate/strong evidence to reject the null hypothesis.
• The linear relationship beween Y and X is significant at 5% level.
• We have weak/moderate/strong evidence of a linear relationship
between Y and X.
• Case 2: Pt > 0.05
.
It doesn't ⇒*
empty that Ho is true .*
• We failed to reject the null. (or, there is not enough evidence to reject
the null.)
• We don’t have evidence to indicate a linear relationship between Y and
X.
4/50
Week 08 - Learning objectives & Outcomes

• Review on matrices.
• What is matrix?
• Matrix: transpose, multiplication, addition, inverse
• Special matrices: identity, diagonal, column vector, row vector.
• Variance-covariance matrix of a random vector.
• Matrix Differentiation.
* • Simple Linear Regression Model in Matrix form.
• SLR model in matrix form.
• Least Squares Estimation of Regression Parameters in matrix form.
• Fitted values and residuals in matrix form.
• Inference in Regression Analysis
• ANOVA in matrix form

5/50
Review on Matrices

6/50
Review on Matrices

• Before we take on multiple linear regression, we first look at simple

regression from a matrix perspective.
• It is easy to generalize it to the general analysis of linear models.

• A matrix is a rectangular array of elements arranged in columns and

rows. 5 6 5 6
10 5 8 a1,1 a1,2 a1,3
A= =
4 12 2 a2,1 a2,2 a2,3

• Matrices are often denoted by captial letters.

• Dimension of a matrix is: (# rows )◊(# columns )
• Elements of matrices are referenced by subscripts (i, j), representing
row and column index. e.g. a1,3 = 8, a2,2 = 12

7/50
Review on Matrices (cont.)

• In general, a matrix can be represented in full or abbreviated form

5 6
a1,1 a1,2 a1,3
A= = A2◊3 = [ai,j ], i = 1, 2; j = 1, 2, 3
a2,1 a2,2 a2,3

• Matrix with 1 column called column vector.

5 6
a
A2◊1 = 1,1
a2,1

• Matrix with 1 row called row vector.

# $
A1◊3 = a1,1 a1,2 a1,3

8/50
Review on Matrices (cont.)

• Matrix transpose: flipped version of matrix.

• Denoted by AÕ , or AT .
5 6
a1,1 a1,2 a1,3
A2◊3 =
a2,1 a2,2 a2,3
C D C D
b1,1 b1,2 a1,1 a2,1
B3◊2 = A = b2,1
Õ
b2,2 = a1,2 a2,2 ,
b3,1 b3,2 a1,3 a2,3
• That is, bi,j = aj,i , j = 1, 2; i = 1, 2, 3
• Transpose of a column vector is a row vector.
5 6
a1,1 # $
A2◊1 = ∆ AÕ = a1,1 a2,1
a2,1

• Matrix equality: two matrices are called equal if they have . same
dimensions and the corresponding elements
. are equal.
• (ABC )T = C T B T AT

9/50
Review on Matrices (cont.)

• Matrix addition: the sum of two matrices with equal dimensions is

matrix of the sum of their corresponding elements.

Am◊n + Bm◊n = Cm◊n … [ci,j ] = [ai,j + bi,j ]

• For example, sum of two column vectors is another column vector

5 6 5 6 5 6
4 2 6
A2◊1 + B2◊1 = + =
6 4 10

• Matrix multiplication: product of two matrices is matrix of

cross-product of their corresponding rows and columns.
n
Ëÿ È
Am◊n ·Bn◊p = Cm◊p … [ci,j ] = ai,k bk,j , ’i = 1, . . . , m, j = 1, . . . , p
k=1
5 6
# $ 1 2 # $ # $
A1◊2 B2◊2 = 1 2 = 1·1+2·3 1·2+2·4 = 7 10
3 4
10/50
Review on Matrices (cont.)
Matrix multiplication:
• # columns for A must match # rows for B.
• Matrix multiplication is NOT symmetric.

A1◊2 · B2◊2 ”= B2◊2 · A1◊2

z EE

• Multiplication by a scalar (number)

5 6 5 65 6
6 2 3ú6 3 ú 2 18 6
c · A2◊2 = 3 · =
7 4 3ú7 3 ú 4 21 12

Diagonal matrix:
• It is a square matrix and all off-diagonal elements are 0.
S T S T
d1 0 0 1 0 0
D3◊3 = U 0 d2 0 V , I3◊3 = U0 1 0V
0 0 d3 0 0 1

• Identity matrix: a special case of diagonal matrix, with all diagonal

elements equal to 1. 11/50
Identity Matrix

• Multiplying matrix A by identity matrix I leaves A unchanged.

• IA=AI=A
S TS T S T
1 0 0 2 4 1 2 4 1
I · A = U0 1 0V U5 1 3 V = U5 1 3V=A·I
0 0 1 7 9 12 7 9 12

• Scalar matrix: diagonal matrix with all diagonal elements being equal
S T S T
d 0 0 1 0 0
U0 d 0 V = d · U0 1 0V = d · I
0 0 d 0 0 1

12/50
Inverse of a Matrix

Assume A is invertible, that is the inverse of A exists.

)

A≠1 A = AA≠1 = I

For a 2 by 2 matrix, a simple formula exists to find its inverse

3 4
a b
if A = ,
c d
3 4
1 d ≠b
then A ≠1
=
ad ≠ bc ≠c a

• Note that the quantity ad ≠ bc is the determinant of A.

• 1/(ad ≠ bc) is not defined when ad ≠ bc = 0 since it is never
possible to divide by zero. It is for this reason that the inverse of A
does not exist if the determinant of A is zero.

13/50
Inverse of a Matrix (cont.)

• Not every square matrix A has an inverse.

• Matrix must be non-singular (full rank).
• Matrix rank is maximum number of linearly indepedent columns.
(equivalently rows).
Az=Ar2A2
C D C D
| | | 3 2 ≠1
A3◊3 = a1 a2 a3 = 2 0 2
| | | 1 1 ≠1

This leads to rank(A)=2, since a3 = a1 ≠ 2a2 .

• Square matrix is full-rank if rank(An◊n ) = n.
• (AB)≠1 = B ≠1 A≠1 assume the square matrices A and B, are both
invertible.
• {(A)≠1 }Õ = {AÕ }≠1

14/50
Column Vectors

• A column with all elements 1 and zero are 1 and 0

S T S T
1 0
W1X W0X
W X W X
1r ◊1 = W . X , 0r ◊1 = W . X
U .. V U .. V
1 0

• A square matrix with all elements 1: J

S T
1 S T
W1X # 1 ... 1
W X $ W .. .. .. X
Jr ◊r = 11T = W . X 1 1 . . . 1 = U. . .V
U .. V
1 ... 1
1

• 1Õ 1 = n

15/50
Mean of a random Vector

• Vecotr of random variables (RV’s)

ST
Y1
WY2 X
W X
Yn◊1 =W . X
U .. V
Yn

where Y1 , . . . , Yn are RV’s.

• Mean of random vector: E (Yi ) = µi , i = 1, . . . n
S T S T
E (Y1 ) µ1
WE (Y2 )X Wµ2 X
W X W X
E (Y)n◊1 = W . X = W . X = µn◊1
U .. V U .. V
E (Yn ) µn

16/50
Mean of a linear combination of Random Vector

• Linear combination of random vectors

• Constant matrix, vector: Ar ◊n , br ◊1
• Random vector: Yn◊1 with E (Y) = µy , Var (Y) = Y.

Xr◊1 = br◊1 + Ar◊n Yn◊1

*V Y d
RT
• For example some constant

S T
5 6 5 6 Y
b a a1,2 a1,3 U 1 V
b2◊1 + A2◊3 Y3◊1 = 1 + 1,1 Y2
b2 a2,1 a2,2 a2,3
Y3
5 6
b1 + a1,1 Y1 + a1,2 Y2 + a1,3 Y3
X2◊1 =
b2 + a2,1 Y1 + a2,2 Y2 + a2,3 Y3
• Mean of X: µx = E (X) = E (b+AY) = b + AE (Y) = b + Aµ
µy

17/50
Var-Cov of Random Vectors

• Variance-covariance matrix of random vector

SY T S V (Y ) Cov (Y1 , Y2 ) ... Cov (Y1 , Yn )
T
1 1
Y 2
W X W Cov (Y 2 , Y 1 ) V (Y2 ) ... Cov (Y2 , Yn )X
Var (Y) = Var (U . V) = U .. .. .. .. V= Y
. . . . . .
Yn Cov (Yn , Y1 ) Cov (Yn , Y2 ) ... Var (Yn ) nxn
nxl Hady

µ
mat of F

Eye
var -

an

• Variance-covariance matrix is symmetric

• Cov (Yi , Yj ) = Cov (Yj , Yi ), that is = T 0 on diagonal : var Mil

• variance-covariance for ‘ off diagonal :

ioytljitjl
Var (‘‘) = ‡ 2 I
If
3

L
=
Ty
1
52 = v th )
.

-
.
. .
= VCYN )

{ constant

CMYI ,Yj
variance

errors
uncorrelated
@ )i±j=o are
:

18/50
Var-Cov of Random Vectors
Show Var (Y) = E {[Y ≠ µ] · [Y ≠ µ]Õ }

Var (Y) = Y = E {[Y ≠ µ] · [Y ≠ µ]Õ }

SY ≠ µ T
1 1

WY2 ≠ µ2 X #Y ≠ µ Y 2 ≠ µ2
$
Y n ≠ µn )
= E (U .. V 1 1 ... nxn ⇒ matrix
. Hh

Y n ≠ µn nxl
S (Y1 ≠ µ1 )2 (Y1 ≠ µ1 )(Y2 ≠ µ2 ) ... (Y1 ≠ µ1 )(Yn ≠ µn )
T
W .. .. .. .. X
= E (U . . . . V)
..
(Yn ≠ µn )(Y1 ≠ µ1 ) (Yn ≠ µn )(Y2 ≠ µ2 ) . (Yn ≠ µn )2

S
V (Y1 ) Cov (Y1 , Y2 ) ...
T
Cov (Y1 , Yn )
W Cov (Y2 , Y1 ) V (Y2 ) ... Cov (Y2 , Yn )X
=U .. .. .. .. V
. . . .
Cov (Yn , Y1 ) Cov (Yn , Y2 ) ... Var (Yn )
= n◊n

19/50
Var-Cov Random Vectors

• Variance-covariance matrix for uncorrelated or independent vector of

RV’s
• Y1 , . . . , Yn with Cov (Yi , Yj ) = 0, ’i ”= j.
S T
‡12 0 ... 0
W0 ‡22 ... 0X
Var (Y) = Y =W
U ... .. .. .. XV
. . .
2
0 0 ... ‡n

• It is a diagonal matrix: all off-diagonal elements are 0.

• W, Y: random vectors
• A: a constant matrix
• We then have
• E(A)=A
• W=AY, E(W)=E(AY)=AE(Y)
• W=AY, Var(W)=Var(AY)=AV (Y )AT *

20/50
Example: Var-Cov of linear combination of random vector
• Let X = b+AY.
• Using (A + B)C = AB + AC , (AB)T = B T AT ,
• Show the var-cov matrix of X is
X = E {(X ≠ µx )(X ≠ µx )T } = A yA
T # I

proof :
.
x=bt AT

EHKECBTAYI = bt AEIY )

lbt AELY) Al Ytly )

=(btAY) AER ) ) AY
=

X Ehl
-

° - - =

Now by defh
}
AHMHTLHLMYIT
El
Excel l*µ×)L*µ×T ) =

zi
: CABIEBTA
'

=
} Ef AH .

My )tfµy
a-
)
T
AT

=
A E{ H-µy ) (Y -

µyM } AT

=AZy AT
÷
a. E. D .

21/50
Example 2 on Var-Cov .
Yi~H( 0 ,
Try

{ Yz NLO Tie )
.

6 ⇒ imply
~
,

5 6 5 6 5 2 only .k)=Jnz
Y1 0
.

‡1 ‡1,2
≥ N( , 2 ), Y ≥ N(0, )
Y2 0 ‡1,2 ‡2
Show V (a1 Y1 + a2 Y2 ) = a12 ‡ 2 + a2 ‡22 + 2a1 a2 ‡1,2

Proof : Method 1 :

vcaihtaihl =
vcaihltvcaihltscwlaia ,
ask )
=
airptairitzaiaz Couch ,k )
=
aprptazrit 29929,2
r#
Method 2 :

W=aihtazYz= ( a , as )l¥z ) =
Anika

Varlwl Van AY ) by * ) slide 21

⇒ A Ey AT
-

= =

=
ca , an 1%2
EI
.
's
) ( Aa )
%
=
atop taint 29,925,2
d. E. D.

22/50
Example 3 on Var-Cov
• Example: Y1 , Y2 ≥ N(0, 1) independently.
• Show
5 6 5 65 6 5 6 5 6
W1 1  0 Y1 0 1 fl
W= = ≥ N( , )
W2 fl 1 ≠ fl2 Y2 0 fl 1
= T
proof : W AY and µy= It9)
=
, 181 I =
=L
.
,

•
Elw ) =
EIAT ) =
A Ely) =
02×1

var ( w
Ee AT
) Var ( AT )
.

=
= A

=
AIAT =AA7

Itif ) tfp
'
1 )
=

.
o .

=
( f ) QE .
D .

23/50
Matrix Differentiation

Y = f (X ), where xn◊1
m◊1

Then the m ◊ n matrix of first-order partial derivatives of the

transformation from x to Y:
S ˆY1 T S ˆy1 ˆy1 ˆy1
T
ˆX ˆx 1 ˆx 2
. . . ˆx n
W ˆy2 ˆy2 ˆy2 X
ˆY W
ˆY2 X
W ˆX X W . . . ˆx X
=W . X=W
ˆx1 ˆx2 n X

.
ˆX U . V U .. W . .
.. .
.. .
.. XV
ˆYn ˆym ˆym ˆym
X . . .
ˆX ˆx1 ˆx2 ˆxn m◊n

24/50
Matrix Differentiation (cont.)

ˆY
MD01 : Y = A X ∆ =A
m◊1 m◊n n◊1 ˆX
Proof:

Hint take

Efta
In

.it?aInMIt*
⇒ Yi =
# AIKXK

⇒
Thing and
=
aij Hit 2 m j⇒ 2 n
. -

,
.
.

, ,
,

2 Yi
=)
Tx
=
[ Air Aiz Ain ]
- -
-

, ,

⇒
QED .

25/50
Matrix Differentiation (cont.)

MD02 : – = Y T
1◊1
A
1◊m m◊n n◊1
X ∆ OM20L
ˆ–
ˆX
= Y T A, and
ˆ–
ˆY 4
= X T AT

Proof:
@ WEYTA ⇒ 2= WTX

⇒
¥× = WT by MDOI

←
-

=
YTA

× , ⇒ 2=51 =
XTATY

⇒ 22
XTAT
Ty
=

a. ED .

26/50
Matrix Differentiation (cont.)
For the special case in which the scalar – is given by the quadratic form
ˆ–
MD03 : – = X T A X ∆ = X T (A + AT )
1◊1 1◊n n◊n n◊1 ˆX
Proof:
An An
-
-

Ain

by defn 2=1×1×2 ,
-
'

xnl ( ian ain) ( ¥xn )

, an ,
. .

j= EYE
! aij
L - ,
Xi
Xj

2¥ #
⇒

.
=
# , akjxj t Aik Xi

=
ix. x. -
-

xn ) Makin ) + ix. .
.

xn ) µa¥m )
,

=
XT Artowk t XTA column k for Kb 2
,
- -

n
,

--
'

⇒
2¥ = XTAT + XTA =
XTIATTAI
27/50
Matrix Differentiation (cont.)
For the special case in which the scalar – is given by the quadratic form,
and the A is a symmetric matrix, ⇒ A=AT
ˆ–
MD04 : – = X T A X ∆ = 2X T A
1◊1 1◊n n◊n n◊1 ˆX
Proof:

From MD03 we have

¥ = XTIATAT )

=XT ( ATA ) since A=A7

=2XTA

QE . D .

28/50
Matrix approach to SLR

29/50
SLR in matrix form

• Y: consisting of the n observations on the response variables.

S T
Y1
WY2 X
W X
Yn◊1 = W . X
U .. V
Yn

• X Matrix: S T
1 X1
W1 X2 X
W X
Xn◊2 = W. .. X
U .. . V
1 Xn
often referred to as the design matrix.

30/50
SLR in matrix form

,¥x=
iIn .tn?i&xntItExiFIil
ix. !
'

x.

• Product

/
,
n
ÿ
= Mi .
I -
-

Ynl (
l¥n ) =
Fun Yj2

YT Y = Yi2 : a scalar, a sum of square terms

1◊1
" i
5 q 6
i 2◊1
X T
Y = q Yi
(Xi Yi )
ixixi
: = 'xnt×t¥nkit¥Ixi% )
5 q 6
, T n
X X= q q X2i detcxtx ) NEXT )2
fix
Xi Xi ⇒
-

1◊1
=

S T
Exitnxy
2 =a
1 =n(
+ q(XX̄ ≠X̄ )2 ≠ q(XX̄≠X̄ )2
"

n
∆ (XT X)≠1 = U i i
V =n Ein lxixt
1◊1 ≠ q(XX̄≠X̄ )2 q 1 2
(X ≠X̄ )
,

i i
=nS××

f- Exit
mm

=
#IIENKITY
Ali #
= H÷nm¥xtnItnt¥*
31/50
SLR in matrix form

• Simple linear regression: mean response in matrix form

E (Yi ) = —0 + —1 Xi , i = 1, . . . , n

• Let
S T S T
E (Y1 ) 1 X1
WE (Y2 )X W1 X2 X 5 6
W X W X —0
E (Y)n◊1 = W . X , Xn◊2 = W . .. X , — 2◊1 = —
U .. V U .. . V 1
E (Yn ) 1 Xn

• Columns represent variables.

• Rows represent observations.

32/50
SLR in matrix form (cont.)

• SLR: Mean response in matrix form

S T S T S T
E (Y1 ) 1 X1 —0 + —1 X1
WE (Y2 )X W1 X2 X 5 6 W— + — X X
W X W X —0 W 0 1 2X
E (Y)n◊1 = W . X = Xn◊2 ·— —2◊1 = W . X =W . X
U .. V U .. V —1 U .. V
E (Yn ) 1 Xn —0 + —1 Xn

33/50
SLR model in matrix form (cont.)

• Simple linear regression in matrix form

Yi = —0 + —1 Xi + ‘i , i = 1, . . . , n

Yn◊1 = Xn◊2— 2◊1 + ‘ n◊1

That is S T S T S T
Y1 1 X1 ‘1
WY2 X W1 X2 X 5 6 W‘ X
W X W X —0 W 2X
W .. X = W .. X +W.X
U . V U. V —1 U .. V
Yn 1 Xn ‘n
• For Y, X
• Columns represent variables.
• Rows represent observations.

34/50
SLR model in matrix form (cont.)

• The normal error regression model

Y = X—
— +‘

• ‘: a vector of independent normal r.v.

• E (‘‘) = 0
• V (‘‘) = ‡ 2 I
• Above 3 conditions on the error term can be written as
S T S T
0 1 0 ... 0
W . X 2 W. .. .. .. X
‘ ≥ N(00, ) = N(U .. V , ‡ U .. . . . V)
0 0 0 ... 1

or in a very brief way, ‘ ≥ N(0, ‡ 2 I)

35/50
Least Squares Estimation of Regression Parameters

• Normal Equations

ÿ ÿ
nb0 + b1 Xi = Yi (1)
ÿ ÿ ÿ
b0 Xi + b1 Xi2 = Xi Yi (2)

This is equivalent in matrix form

X ÕX b = X ÕY
2◊2 2◊1 2◊1
5 q 65 6 5 q 6
n X i b0 Y i
X X ·b = q
Õ q 2 =X Y = q
Õ
Xi Xi b1 Xi Yi
• The vector of the least squares regression coefficients
S 2
T
1 q X̄ q X̄ 2 5 q
5 6
b + ≠ 6
V q Yi
n (Xi ≠X̄ ) 2 (Xi ≠X̄ )
b = 0 = (XÕ X)≠1 XÕ Y = U
b1 ≠ q X̄ 2 q 1 2 Xi Yi
(Xi ≠X̄ ) (Xi ≠X̄ )

36/50
Verify the LS matrix form Exercise :

S 2
T
1
5 6
b0 + q(XX̄ ≠X̄ )2 ≠ q(XX̄≠X̄ )2 5 q 6
V q Yi
n
b= = (XÕ X)≠1 XÕ Y = U i i

b1 ≠ q X̄ 2 q 1 Xi Yi
(Xi ≠X̄ ) (Xi ≠X̄ )2

et sxx =
-2¥ Ni # 12

±nt¥× ¥x× Emi

.si/y=IlXi-X1lYiT)-fg
-

Elkin ) hit =
'
-

0Elxi-x1lYj-51-TlXiYjtnxjth@Elxiyp-EXj2-nx2proof.L
( )( )
i
-

Fxx s÷× Enxiti

.be#EIIxjIksxIx=nnI+n5gxI.xEhgxiI
5- ¥i¥±# "
5 =b

)= (
.

aximj.IE/=b.=c*bbi
-

*¥× +
age •

's =iIxbY*l 37/50

Least Squares Estimation of Regression Parameters
Minimize the quantity

ya
ÿ

,n×Q
Q= [Yi ≠ (—0 + —1 Xi )]2 B :
2×1
T
= (Y ≠ X —) (Y ≠ X —) ' '' 2 zxnnx
¥ ( BTXTY )
= Y T Y ≠ —T X T Y ≠ Y T X — + —T X T X —
1×1 1×2 lxhnxz =¥p ( YTXP )

"nxiLeIFtexxEf
2×h nxl zx ,

To find the value of — that minimizes Q:

=
YX IMDOZ )
ˆ # $
Q = ˆQ/ˆ—0 ˆQ/ˆ—1 = ≠2Y Õ X + 2— T X T X 2
Fp ( ytxp )
ˆ—
=Y1X
• Setting this equation to zero: ≠2Y Õ X + 2b T X T X = 0 ( MDOZ )
• Transpose both sides of the equation, we have
¥
3

lptxtxp )
X Õ Xb = X Õ Y
ˆ Assume X Õ X is invertible
• Solving it for —. =zpT( XTX )
1 MD 04 )
∆ b = (X Õ X )≠1 X Õ Y
38/50
Fitted Values and Residuals

Fitted Values:

39/50
Fitted Values and Residuals (conts.)

⇐
make.tt?IiiIIiwll?d.hiIItIiIIEEI?I
Hat Matrix:
• H = X (X Õ X )≠1 X Õ : Hat matrix or “projection” matrix. I ,
• Important in diagnostics for regression
• Symmetric and idempotent:
.
Eh hi ; = 1

&
H T = H; HH = H -3¥ hiijihii

1
HE [ xixix , ix. 1 '=×[w× .yt×
,
-

,
hij-hjithEiIhiie2n-xtlxTxPItXT-XlXTX5lXT-H@HH-XlXTX5lXTXcXTX51xT-XcxTxjtXT-Hmn_MHH-HH0EiYhIy-hii0Zkhahikh.k

- ;
=
hij
40/50
Residuals

41/50
Residual distribution

Y ≥ N(X —, ‡ 2 I), ⇒ e ≥ N(0, ‡ 2 (I ≠ H))

n◊1 n◊1
Proof:

e=Y 'T y HY
=
=h HIY where

*
KXHX
'
He
-
-

xitxtxxp
,

⇒ EL HM
f- ) =
H -

HIEN ) 2

xp HXB xp =Xp Xp :O
= -
=
- -

w nm

Van e) Var
=
( EHN )
=
(z -

H) Zy H -

HIT
z .

H is idempotent'

'
= ( z -

His 1 a- H )
x

=P ( THHTHI = 02 I Ht )
42/50
Inference in Regression Analysis

symmetric
ttforvaravmatrixif
Regression Coefficients:
• The variance-covariance matrix of b:

5 6
V (b0 ) V (b0 , b1 ) combo b.) 0-25
Var (b) =
=
-

V (b1 , b0 ) V (b1 ) Sxx

11
S 2
T
2 1 X̄ 2 X̄
Var ( b o
) ⇐ ‡ ( n + q(X ≠X̄ )2 ) ≠‡ q(X ≠X̄ )2
=U i
2
i
V
≠‡ 2 q(XX̄≠X̄ )2 q ‡
(X ≠X̄ ) 2
i i

2
= ‡ (X X )Õ ≠1
"

var ( b
,
)=E÷×

43/50
Inference in Regression Analysis (cont.)

• The estmiated variance-covariance of b:

s 2 (b) = MSE (X Õ X )≠1

2◊2
S 2
T
1
n + q(XX̄ ≠X̄ )2 ≠ q(XX̄≠X̄ )2
= MSE U i i
V
≠ q(XX̄≠X̄ )2 q 1
(Xi ≠X̄ )2
i

2
=‡
ˆ (X X )
Õ ≠1

• b = (X Õ X )≠1 X Õ Y = AY
T
• ∆ V (b) = V (AY ) = A YA = ‡ 2 AAT = ‡ 2 (X Õ X )≠1

44/50
Mean response and prediction of new observation

• The mean response at Xh

5 6
1
Xh = ; Ŷh = XhÕ b
2◊1 Xh

V (Ŷh ) = V (XhÕ b) = XhÕ V (b)Xh

= XhÕ ‡ 2 (X Õ X )≠1 Xh
= ‡ 2 XhÕ (X Õ X )≠1 Xh
1 (Xh ≠ X̄ )2
= ‡2 [ + q ]
n (Xi ≠ X̄ )2
∆ s 2 (Ŷh ) = MSE (XhÕ (X Õ X )≠1 Xh )

• Prediction of New observation at Xh

2
spred = MSE (1 + XhÕ (X Õ X )≠1 Xh )

45/50
Analysis of Variance Results (ANOVA)
Total Sum of Squares
ÿ 1 Õ 1
SSTO = (Yi ≠ Ȳ )2 = YÕ Y ≠ Y JY = YÕ [I ≠ J]Y
n n
Proof:
Il Yi -

F) 2= Eyeing
2
= zy ,
?
-

K¥2
'
th 141

Hk
Y Y )
= '
-

th
' ' '
=
Y Y -

IY 12 Y)
'
=
y y -
th LYJY )

his ) y
'
=
Y (z
-

46/50
Analysis of Variance Results (ANOVA) (cont.)
Error Sum of Squares

SSE = e Õ e = (Y ≠ Xb)Õ (Y ≠ Xb) = YÕ Y ≠ bÕ XÕ Y = YÕ (I ≠ H)Y

g- @-
Proof:
1

sseeeie-H-xbjlY-Xb1-Htb1xYH-xb1-Y4-Yxb-b1XYtbX1xb_ib-lXXMXYb1-YlXCX1X5l-Yy-Yxb-b1XYtYIIkXExxbtYY-YXb-b1XYtYXbae-b1XY@sSEee1e-ly-FpCy-P1-ktHNj4tHjY-YCz-HH2-HM-Y1CtH1YQ.E.D

47/50
Analysis of Variance Results (ANOVA) (cont.)
Regression Sum of Squares
ÿ 1 Õ 1
SSR = (Ŷi ≠ Ȳ )2 = bÕ XÕ Y ≠ Y JY = YÕ [H ≠ J]Y
n n
Proof:

EYIETTY
DssRt-HiYitl2-EIcYitzFItF2l-2EFYY-2Y2iFkTth.lxi-xHtny2.Yi-botkXi-FbiItb.Xi-2iFitit25enytb.Elxixntny2-5tbilxiX1-O-EiIFi2-2nT2tny2l-Ei.FYi2-nYT0EFi2tlxblTcxbkbTXTxb-blXx1llxX5lMyI-b1XY@ny2.t

-n( EYIP =
'T ((141417 ) ) =
th YJY

ssR= 1
-
2 =
b 'X' Y -

th YYY
30-20=4
IH HTTY
'

)=Y' AT
-

HYITCHY
'

3
= ( ⇒ ssr=
48/50
Analysis of Variance Results (ANOVA) (Summary)

• Sums of Squares as Quadraatic Froms: Y Õ AY

ÿ ÿ 1
SSTO = Yi2 ≠ ( Yi )/n = YÕ (I ≠ J)Y
n
SSE = e Õ e = (Y ≠ Xb)Õ (Y ≠ Xb) = YÕ (I ≠ H)Y
ÿ 1
SSR = (Ŷi ≠ Ȳ )2 = YÕ [H ≠ J]Y
n

49/50
Practice problems and upcoming topics

• Practice problems after today’s lecture: CH5: 5.1, 5.3, 5.4, 5.8, 5.10,
5.12, 5.17, 5.21, 5.23, 5.29, 5.31
• Extra exercises
• Show I ≠ H is symmetric and idempotent.
• Show H(I ≠ H) = 0
• Upcoming topics
• CH6: Models with more than one predictors.
• Reading for upcoming topics: Chapter 6.

50/50