Logistic Regression

FSD Team

Informatics, UII

October 9, 2022

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 1 / 10

Linear Regression: the procedure

1 Pick an initial line/model h(✓) by randomly choosing parameter ✓

2 Compute the corresponding cost function, e.g.,
m
1X
J(✓) = (h✓ (x(i) ) y (i) )2 ,
2
i=1

3 Update the line/model h(✓) by updating ✓ that makes J(✓) smaller,

using, e.g., the gradient descent that reads
@
✓j := ✓j ↵ J(✓),
@✓j

where ↵ is the learning rate.

4 Repeat steps 2 and 3 until converges

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 3 / 10

Linear regression: a summary

In more detail
The learning procedure above is a typical approach in mostly (parametric)
models of machine and deep learning.

Note that we have been so far assuming the y in our data set
(x(i) , y (i) ); i = 1, . . . , m is continuous, e.g., house price, height,
temperature, etc.

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 4 / 10

Linear regression: a summary

In more detail
The learning procedure above is a typical approach in mostly (parametric)
models of machine and deep learning.

Note that we have been so far assuming the y in our data set
(x(i) , y (i) ); i = 1, . . . , m is continuous, e.g., house price, height,
temperature, etc.

What if y is discrete? E.g., pass/fail, yes/true, 1/0, etc, that leads to

nonlinear fashion and a classification task.

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 4 / 10

Linear regression: a summary

In more detail
The learning procedure above is a typical approach in mostly (parametric)
models of machine and deep learning.

Note that we have been so far assuming the y in our data set
(x(i) , y (i) ); i = 1, . . . , m is continuous, e.g., house price, height,
temperature, etc.

What if y is discrete? E.g., pass/fail, yes/true, 1/0, etc, that leads to

nonlinear fashion and a classification task.

Let’s see the plot example.

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 4 / 10

If y is a discrete variable

1
Exam status

0
2 4 6 8 10
Study hour

Exam status: 0 (fail), 1 (pass)

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 5 / 10

If y is a discrete variable

1
Do you think the linear
model y = ✓0 + ✓1 ⇥ x will
fit well?
Exam status

0
2 4 6 8 10
Study hour

Exam status: 0 (fail), 1 (pass)

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 5 / 10

If y is a discrete variable

Do you think the linear

1 model y = ✓0 + ✓1 ⇤ x will fit
well?
Exam status

Let’s try one.

0
2 4 6 8 10
Study hour

Exam status: 0 (fail), 1 (pass)

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 6 / 10

If y is a discrete variable

Do you think the linear

model y = ✓0 + ✓1 ⇥ x will
1
fit well?
Let’s try one
Exam status

Looks bad, try another one.

0
2 4 6 8 10
Study hour

Exam status: 0 (fail), 1 (pass)

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 7 / 10

If y is a discrete variable

Do you think the linear

model y = ✓0 + ✓1 ⇥ x will
1
fit well?
Let’s try one
Exam status

Looks bad, try another one.

We see that such a linear
model won’t fit well, because
0 of the nonlinear fashion. We
2 4 6 8 10 need another model!
Study hour

Exam status: 0 (fail), 1 (pass)

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 8 / 10

If y is a discrete variable

We can use the logistic

(sigmoid) function that
1
reads
1
h✓ (x) = g(✓T x) =
Exam status

1+e ✓T x

We typically predict “1” if

h✓ (x) 0.5
0
2 4 6 8 10
Study hour

Exam status: 0 (fail), 1 (pass)

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 9 / 10

Logistic Regression: the procedure

1 Pick an initial line/model h(✓) by randomly choosing parameter ✓

2 Compute the corresponding cost function, e.g.,
m
X
J(✓) = y log h✓ (x(i) ) (1 y) log(1 h✓ (x(i) ))
i=1

3 Update the line/model h(✓) by updating ✓ that makes J(✓) smaller,

using, e.g., the gradient descent that reads
@
✓j := ✓j ↵ J(✓),
@✓j

where ↵ is the learning rate.

4 Repeat steps 2 and 3 until converges

FSD Team (Informatics, UII) Logistic Regression October 9, 2022 10 / 10