12/6/2022

REGRESSION
ANALYSIS
WITH
LINEAR PROBABILITY MODEL, LOGIT AND PROBIT MODELS

HELLO!
I am Elijah Appiah from
Ghana.
I am an Economist by
profession.
I love everything about R!

You can reach me:

secret behind the smile! eappiah.uew@gmail.com

1
 12/6/2022

Lesson Goals
▸ Understand how to estimate
qualitative response models.

▸ Advance your knowledge of

statistical inference for limited
dependent variables.

Lecture Series

2
 12/6/2022

Limited Dependent Variables

Sometimes, the dependent variable may be
categorical.

Two categories – binary or dichotomous

Three categories – trichotomous
Four or more categories - polychotomous

Binary Response Variable

The dependent (or response) variable is binary.
Independent variable(s) may be quantitative,
qualitative, or mixed as usual.
Examples of binary variables:
Yes or No responses (0 - No, 1 - Yes)
Patient Satisfaction(0 - Not Satisfied vs. 1- Satisfied)
Health Outcome (0 - Not Cured vs. 1- Cured)
Votes (0 - Democrat vs. 1 - Republican)

3
 12/6/2022

Binary Response Variable

Three approaches to measuring binary response
models:

Linear Probability Model (LPM)

Logistic Regression (or Logit) Model

Probit (or Normit) Model

LINEAR
PROBABILITY
MODEL

4
 12/6/2022

Linear Probability Model

LPM is estimated using Ordinary Least Squares
(OLS).

Coded values, 0 and 1, of binary response

variables are treated as numbers, not categories.

10

Linear Probability Model

Consider the regression model:
𝒊 𝟎 𝟏 𝒊
where: 𝑿𝒊 = family income, 𝒀𝒊 = 𝟏 if the family owns
a house and 𝟎 if it does not own a house.
Conditional expectation - 𝑬(𝒀𝒊 | 𝑿𝒊 ) – is the
conditional probability that the event will occur
given 𝑿𝒊 , that is, 𝑷𝒓(𝒀𝒊 = 𝟏|𝑿𝒊 ).
𝑷𝒓(𝒀𝒊 = 𝟏|𝑿) = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
Hence, Linear Probability Model.

5
 12/6/2022

11

Linear Probability Model

If 𝑷𝒊 is probability that 𝒀𝒊 = 𝟏, and (𝟏 − 𝑷𝒊 ) is
probability that 𝒀𝒊 = 𝟎, then the variable 𝒀𝒊 will
have the probability distribution as:
𝒀𝒊 Probability

Bernoulli 0 𝟏 − 𝑷𝒊
Probability
Distribution 1 𝑷𝒊
Total 𝟏

12

Linear Probability Model - Problems

Recall: Rule of Probability ~ 𝒊 𝒊 .

Real problem with OLS estimation of LPM is

nonfulfillment of 𝟎 ≤ 𝑬(𝒀𝒊 |𝑿𝒊 ) ≤ 𝟏. That is,
predicted probabilities can be negative, or greater
than 1.
Heteroscedasticity (can be solved using robust
standard errors).
Low R-squared value.

6
 12/6/2022

13

Now, let’s practice

14

Linear Probability Model

𝐏𝐫(𝒀 > 𝟏)

𝐏𝐫(𝒀 < 𝟎)

7
 12/6/2022

15

Linear Probability Model - Problems

Probability Problem (How do we get around this problem?)
Estimate the LPM by the usual OLS
method. If estimated Y has some values
less than 0 [i.e. negative], Y is assumed Constrained LPM
to be zero for those cases; if they are
greater than 1, they are assumed to be 1.

Devise an estimating technique that will

Logit and Probit Models guarantee that the estimated conditional
probabilities Y will lie between 0 and 1.

16

Now, let’s practice

8
 12/6/2022

17

MODEL

18

Logistic Regression or Logit Model

Cumulative plot.ecdf(rnorm(500))
distribution functions
(CDF) are sigmoid, or
S-shaped curves.

Some CDFs are the

logistic CDF (logit) and
standard normal CDF
(probit).

9
 12/6/2022

19

Logit Model
match match frequency What can we
Win
Win 4 do with this
Win
Lose 6 count data?
Win
Win
Lose
Lose What is the probability that a
Lose team wins the match?
Lose
Lose What are the odds of winning?
Lose

20

Logit Model
Probability is the number of events divided by total
outcome.

match frequency 𝟒
𝒑𝒓𝒐𝒃 = = 𝟎. 𝟒𝟎
Win 4 𝟏𝟎
Lose 6

Odds are the chances of something happening to

something not happening.
𝟒
𝒐𝒅𝒅𝒔 = = 𝟎. 𝟔𝟕
𝟔

10
 12/6/2022

21

Logit Model
Probability and odds can be expressed in terms of each other.
𝒐𝒅𝒅𝒔 𝒑𝒓𝒐𝒃
𝒑𝒓𝒐𝒃 = 𝒐𝒅𝒅𝒔 =
𝟏 + 𝒐𝒅𝒅𝒔 𝟏 − 𝒑𝒓𝒐𝒃

𝟒 𝟒
𝒑𝒓𝒐𝒃 = = 𝟎. 𝟒𝟎 𝒐𝒅𝒅𝒔 = = 𝟎. 𝟔𝟕
𝟏𝟎 𝟔

𝟎. 𝟔𝟕 𝟎. 𝟒𝟎
𝒑𝒓𝒐𝒃 = = 𝟎. 𝟒𝟎 𝒐𝒅𝒅𝒔 = = 𝟎. 𝟔𝟕
𝟏 + 𝟎. 𝟔𝟕 𝟏 − 𝟎. 𝟒𝟎

22

Logit Model
Logistic regression (logit) model calculates the
log odds given known values of the predictors.
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊

11
 12/6/2022

23

Logit Model
𝒐𝒅𝒅𝒔
Recall: 𝒑𝒓𝒐𝒃 =
𝟏 + 𝒐𝒅𝒅𝒔
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊

𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Not Correct Yet!
𝟏 + 𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)

Exponential function? ----> 𝒇(𝒙)

𝒍𝒏[𝒇(𝒙)] 𝒍𝒏 𝒙 𝒍𝒏 𝟐

24

Logit Model
𝒐𝒅𝒅𝒔
Recall: 𝒑𝒓𝒐𝒃 =
𝟏 + 𝒐𝒅𝒅𝒔
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊

𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Not Correct Yet!
𝟏 + 𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)

𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Correct
𝟏 + 𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)

12
 12/6/2022

25

Logit Model
𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Correct
𝟏 + 𝒆𝒍𝒐𝒈(𝒐𝒅𝒅𝒔)
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊

𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 =
𝟏 + 𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊

𝟏
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = (𝜷𝟎 𝜷𝟏 𝑿𝒊 )
𝟏+𝒆

26

Logit Model
𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 =
𝟏 + 𝒆 𝜷 𝟎 𝜷 𝟏 𝑿𝒊
𝜷𝟎 𝜷𝟏 𝑿𝒊
𝜷𝟎 𝜷𝟏 𝑿𝒊
𝒊 𝒊 𝜷𝟎 𝜷𝟏 𝑿𝒊
𝜷𝟎 𝜷𝟏 𝑿𝒊 𝜷𝟎 𝜷𝟏 𝑿𝒊
𝟏
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = (𝜷𝟎 𝜷𝟏 𝑿𝒊 )
𝟏+𝒆

13
 12/6/2022

27

Logit Model
𝐥𝐨𝐠 𝒐𝒅𝒅𝒔 𝑬 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
Probability is not
𝒁𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊 linear in 𝑿𝒊 or the
parameters in 𝒁𝒊 .

𝒆𝒁𝒊
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = Logistic
𝟏 + 𝒆𝒁𝒊
Distribution
𝟏 Function
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝒁𝒊
𝟏+𝒆

28

Now, let’s practice

14
 12/6/2022

29

Logit Model
𝟏
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝒁𝒊
𝟏+𝒆
𝟏 𝟏 𝟏
𝑷𝒊 = 𝒁𝒊 𝟏 − 𝑷𝒊 = 𝟏 − 𝒁𝒊 =
𝟏+𝒆 𝟏+𝒆 𝟏 + 𝒆𝒁𝒊
𝒑𝒓𝒐𝒃
𝒐𝒅𝒅𝒔 =
𝟏 − 𝒑𝒓𝒐𝒃
𝒊 𝒁𝒊
𝒁𝒊
𝒊
𝒁𝒊 Odds Ratio

30

Logit Model
𝒑𝒓𝒐𝒃
𝒐𝒅𝒅𝒔 =
𝟏 − 𝒑𝒓𝒐𝒃
𝒊 𝒁𝒊
𝒁𝒊
𝒊
𝒁𝒊 Odds Ratio

If ,
The odds that the family
will own a house is 4 to 1.

15
 12/6/2022

31

Logit Model
𝑷𝒊
𝑶𝒅𝒅𝒔 = = 𝒆𝒁𝒊
𝒙 𝟏 − 𝑷𝒊

𝑷𝒊 𝑷𝒊 𝒁
𝑶𝒅𝒅𝒔 = 𝒍𝒏
𝑳 = 𝒍𝒏(𝑶𝒅𝒅𝒔) = 𝒆 𝒊 = 𝒍𝒏 𝒆𝒁𝒊 = 𝒁𝒊
𝟏 − 𝑷𝒊 𝑷𝒊
𝟏 −

𝑳 = 𝒁𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊
where: 𝑳, is the log of the odds ratio, is not only
linear in 𝑿𝒊 but also the parameters. L is called
the Logit, hence the name Logit model.

32

Logit Model
Key Takeaways:
The logits are not bounded by 0 and 1, but the
probabilities will (so we need to calculate
probabilities for the logit model – MARGINAL
EFFECTS).
Although L is linear in X, but the probabilities are
not.
You can include many predictors.

16
 12/6/2022

33

Logit Model
Key Takeaways:
If L, the logit, is positive, it means that when the
value of the regressor(s) increases, the odds that
the regressand equals 1 (meaning some event of
interest happens) increases. If L is negative, the
odds that the regressand equals 1 decreases as
the value of X increases.

34

MODEL

17
 12/6/2022

35

Probit Model
The probit model uses a standardized normal CDF.
If a variable 𝑿 follows the normal distribution with
mean 𝝁 and standard deviation 𝝈, then the PDF is:
𝟏 𝟐 𝟐
𝒇 𝑿 = 𝒆 𝑿 𝝁 /𝟐𝝈
𝟐𝝈𝟐 𝝅

Its CDF becomes:

𝑿𝟎
𝟏 𝑿 𝝁 𝟐 /𝟐𝝈𝟐
𝑭 𝑿 = 𝒆
𝟐𝝈𝟐 𝝅

36

Probit Model
The probit model becomes:
𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝑭(𝜷𝟎 + 𝜷𝟏 𝑿𝒊 )

Thus, if 𝒁 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊 , then the CDF is:

𝜷𝟎 𝜷𝟏 𝑿𝒊
𝟏 𝒁𝟐 /𝟐
𝑭 𝑰 = 𝒆 𝒅𝒁
𝟐𝝅

18
 12/6/2022

37

Logit and Probit Models

Key Notes:
Both models estimate the log odds, but logit
follows a logistic CDF whiles probit follows a
standard normal CDF.
To express coefficients in probabilities, we
calculate the marginal effects.
Both models have predicted probabilities lying
between 0 and 1 inclusive.

38

Logit and Probit Models

Key Notes:
Both models are estimated using an iterative
algorithm to calculate the log odds of the
regression coefficients.

Maximum Likelihood Estimation

19
 12/6/2022

39

Logit and Probit Models - R

Logit and Probit estimates in log odds:

stat::glm()

Logit and Probit estimates in probabilities:

mfx:logitmfx()
mfx::probitmfx()

40

Logit and Probit Models - R

Goodness of Fit:
𝒅𝒆𝒗𝒊𝒂𝒏𝒄𝒆
𝑷𝒔𝒆𝒖𝒅𝒐 𝑹𝟐 = 𝟏 −
𝒏𝒖𝒍𝒍. 𝒅𝒆𝒗𝒊𝒂𝒏𝒄𝒆
deviance and null.deviance are attributes of
both models.

𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒊𝒐𝒏𝒔

𝑪𝒐𝒖𝒏𝒕 𝑹𝟐 =
𝑻𝒐𝒕𝒂𝒍 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒐𝒃𝒔𝒆𝒓𝒗𝒂𝒕𝒊𝒐𝒏𝒔

20
 12/6/2022

41

Now, let’s practice

42

THANKS!
Any questions?

Reach me anytime!
Email
eappiah.uew@gmail.com
LinkedIn
https://www.linkedin.com/in/appiah-elijah-383231123/

21