Linear Regression Introduction

Big Data and Machine Learning for Applied Economics

Ignacio Sarmiento-Barbieri

Universidad de los Andes

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 0 / 17

Agenda

1 Intro

2 Linear Regression Model

3 Statistical Properties

4 Asymptotic Tests

5 Further Readings

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 1 / 17

1 Intro

2 Linear Regression Model

3 Statistical Properties

4 Asymptotic Tests

5 Further Readings

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 1 / 17

Intro

I Linear regression is the “work horse” of econometrics and (supervised) machine

learning.
I Very powerful in many contexts.
I Big ‘payday’ to study this model in detail.

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 2 / 17

Intro

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 3 / 17

1 Intro

2 Linear Regression Model

3 Statistical Properties

4 Asymptotic Tests

5 Further Readings

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 3 / 17

Linear Regression Model

I If f (X) = Xβ, obtaining f (.) boils down to obtaining β

y = Xβ + u (1)

I where
I y is a vector n × 1 with typical element yi
I X is a matrix n × k
I Note that we can represent it as a column vector X = [ X1 X2 . . . Xk ]
n×k n×1 n×1 n×1
I β is a vector k × 1 with typical element β j

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 4 / 17

Linear Regression Model
For example, we have the following data and want to estimate a linear model:

80

70

y
60

50

4 6 8 10
X1

We would specify:

y = β 0 + β 1 X1 + u (2)

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 5 / 17

Linear Regression Model

I How do we obtain β?
I Method of Moments H.W.
I MLE (more on this later)
I OLS: minimize SSR (e0 e)
I where e = Y − Ŷ = Y − X β̂

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 6 / 17

OLS
How do we obtain β?
I Consider the following risk function, where we minimize the sum of square residuals
n
SSR( β̃) ≡ ∑ ẽ2i = ẽ0 ẽ = (y − X β̃)0 (y − X β̃) (3)
i=1

I SSR( β̃) is the aggregation of squared errors if we choose β̃ as an estimator.

I The least squares estimator β̂ will be

β̂ = argmin SSR( β̃) (4)

β̃

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 7 / 17

OLS

SSR( β̃) = ẽ0 ẽ (5)

= (Y − X β̃)0 (y − X β̃) (6)

I FOC are

∂ẽ0 ẽ
=0 (7)
∂ β̃
∂ẽ0 ẽ
= − 2X0 y + 2X0 X β̃ = 0 (8)
∂ β̃

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 8 / 17

OLS

I Let β̂ be the solution.Then β̂ satisfies the following normal equation

X0 X β̂ = X0 y (9)

I If the inverse of X0 X exists, then

β̂ = (X0 X)−1 X0 y (10)

I Note that this is a closed solution (a bonus!!)

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 9 / 17

Assessing the Accuracy of the Coefficient Estimates
I Predicting well in this context → estimating well. Why?

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 10 / 17

Assessing the Accuracy of the Coefficient Estimates
I Predicting well in this context → estimating well. Why?
I The prediction of y will be given by ŷ = X β̂

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 10 / 17

Assessing the Accuracy of the Coefficient Estimates
I Predicting well in this context → estimating well. Why?
I The prediction of y will be given by ŷ = X β̂
80

70

y
60

50

4 6 8 10
X1

I In our simple example the prediction of y will be given by ŷ = β̂ 0 + β̂ 1 X1

I A natural question is how accurate are β̂ 0 and β̂ 1 as an estimate of β 0 and β 1 ?

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 10 / 17

Assessing the Accuracy of the Coefficient Estimates

I The variance of β̂ 0

x̄2
 
2 1
Var( β̂ 0 ) = σ + n (11)
n ∑i=1 (xi − x̄)2

I and β̂ 1

σ2
Var( β̂ 1 ) = (12)
∑ni=1 (xi − x̄)2

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 11 / 17

Agenda

1 Intro

2 Linear Regression Model

3 Statistical Properties

4 Asymptotic Tests

5 Further Readings

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 11 / 17

Statistical Properties

Under certain assumptions HW Review the Assumption from Econometrics

I Small Sample (Gauss-Markov Theorem)

I Unbiased: E( β̂) = β
I Minimum Variance: Var( β̃) − Var( β̂) is positive semidefinite matrix Proof: HW. Remember: a matrix
Mp×p is positive semi-definite iff c0 Mc ≥ 0 ∀c ∈ Rp

I Large Sample
I Consistency: β̂ →p β
√
I Asymptotically Normal: N ( β̂ − β) ∼a N (0, S)

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 12 / 17

Gauss Markov Theorem
I Gauss Markov Theorem that says OLS is BLUE is perhaps one of the most famous
results in statistics.
I E( β̂) = β
I Var( β̂) = σ2 (X0 X)−1

I and implies that ŷ is an unbiased predictor and minimum variance, from the class of
unbiased linear predictors (BLUP) H.W. proof

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 13 / 17

Gauss Markov Theorem
I Gauss Markov Theorem that says OLS is BLUE is perhaps one of the most famous
results in statistics.
I E( β̂) = β
I Var( β̂) = σ2 (X0 X)−1

I and implies that ŷ is an unbiased predictor and minimum variance, from the class of
unbiased linear predictors (BLUP) H.W. proof
I However, it is essential to note the limitations of the theorem.
I Correctly specified with exogenous Xs,
I The term error is homoscedastic
I No serial correlation.
I Nothing about the OLS estimator being the more efficient than any other estimator one can
imagine.

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 13 / 17

1 Intro

2 Linear Regression Model

3 Statistical Properties

4 Asymptotic Tests

5 Further Readings

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 13 / 17

Asymptotic Tests

I Continuing with our simple example

y = β 0 + β 1 X1 + u (13)

I Suppose that you want to test if there’s is no relationship between X1 and y

I The large sample property + the assumptions needed to get there allows us to have
valid tests

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 14 / 17

Asymptotic Tests
I Mathematically testing that there’s is no relationship between X1 and y corresponds
to testing

H0 = β 1 = 0 (14)

H1 = β 1 6= 0 (15)

I and use a t − statistic

β̂ 1 − 0
t= (16)
SE( β̂ 1 )

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 15 / 17

Asymptotic Tests

I However, in some cases the asymptotic approximation need not be very good
I Especially with highly nonlinear models
I Resampling methods can allow us to somewhat quantify uncertainty and improve
on the asymptotic distribution approximations
I Bootstrapping, which is a popular resampling method, can be used as an alternative
to asymptotic approximations for obtaining standard errors, confidence intervals,
and p-values for test statistics.

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 16 / 17

1 Intro

2 Linear Regression Model

3 Statistical Properties

4 Asymptotic Tests

5 Further Readings

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 16 / 17

Further Readings

I Davidson, R., & MacKinnon, J. G. (2004). Econometric theory and methods (Vol. 5). New York: Oxford
University Press.

I James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112,
p. 18). New York: springer.

Sarmiento-Barbieri (Uniandes) Linear Regression Introduction 17 / 17