11/9/22

BENE 4333
ARTIFICIAL INTELLIGENCE

By:
Wira Hidayat bin Mohd Saad
(PhD)

Logistic Regression

1
 11/9/22

Classification
• Email: Spam / Not Spam?
• Online Transactions: Fraudulent (Yes / No)?
• Tumor: Malignant / Benign ?

0: “Negative Class” (e.g., benign tumor)

1: “Positive Class” (e.g., malignant tumor)

Slide credit: Andrew Ng

Classification
hθ(x)
(Yes) 1

Malignant ?
0.5

(No) 0
Tumor Size Tumor Size

Threshold classifier output at 0.5:

If , predict “y = 1”
If , predict “y = 0”

Slide credit: Andrew Ng

2
 11/9/22

Classification
hθ(x)
(Yes) 1

Malignant ?
0.5

(No) 0
Tumor Size Tumor Size

Threshold classifier output at 0.5:

If , predict “y = 1”
If , predict “y = 0”

Slide credit: Andrew Ng

Classification

Logistic Regression: Logistic regression is a classification algorithm

used to assign observations to a discrete set of classes. Unlike linear
regression which outputs continuous number values, logistic
regression transforms its output using the logistic sigmoid function
to return a probability value which can then be mapped to two or
more discrete classes.

3
 11/9/22

Classification
Logistic Regression Model

1
ℎ! 𝑥 = 𝑔 𝜃 " 𝑥 → 𝑔 𝑧 = ………4.1
1 + 𝑒 #$
1
𝑔 𝑧
Sigmoid function = Logistic function
0.5

0 z

Slide credit: Andrew Ng

Interpretation of Hypothesis Output

= estimated probability that y = 1 on input x

Example: If

Tell patient that 70% chance of tumor being malignant

“probability that y = 1, with

given x, parameterized by ”

Slide credit: Andrew Ng

4
 11/9/22

Decision boundary
1
𝑔 𝑧
0.5

Suppose predict “ “ if 0 z
(𝜃 ! 𝑥 < 0) (𝜃 ! 𝑥 ≥ 0)
(𝜃 ! 𝑥 ≥ 0)

predict “ “ if
(𝜃 ! 𝑥 < 0)

Slide credit: Andrew Ng

Decision boundary
x2

2 Let’s 𝜃" = −3, 𝜃# = 1, 𝜃$ = 1,

1
Predict “ “ if
1 2 3 x1
Or in other word, Predict “ “ if
𝑥% + 𝑥& ≥ 3

Slide credit: Andrew Ng

10

5
 11/9/22

Decision boundary
x2

2 Let’s 𝜃" = −3, 𝜃# = 1, 𝜃$ = 1,

1
Predict “ “ if
1 2 3 x1
Or in other word, Predict “ “ if
Decision boundary 𝑥% + 𝑥& ≥ 3

Slide credit: Andrew Ng

11

Non-linear decision boundaries

x2

1
…..(4.2)
-1 1 x1
-1
Let’s 𝜃" = −1, 𝜃# = 0, 𝜃$ = 0, 𝜃% = 1, 𝜃& = 1,

Predict “ “ if
x2
If 𝑥%& + 𝑥&& = 1 , we can draw the circle.

x1

Slide credit: Andrew Ng

12

6
 11/9/22

Cost Function

Training set:

m examples

How to choose parameters ?

Slide credit: Andrew Ng

13

Cost Function
Linear regression cost function :

Logistic regression cost function

Slide credit: Andrew Ng

14

7
 11/9/22

Cost Function

If y = 1 If y = 1,
q Cost = 0 if prediction is correct
Cost
q As ℎ! 𝑥 → 0, cost → ∞
q Captures intuition if ℎ! 𝑥 =0,
but y = 1, we will penalize the
0 1 learning algorithm by a very large
cost.
Slide credit: Andrew Ng

15

Cost Function

If y = 1 If y = 0
If y = 0
q Cost = 0 if prediction is correct
Cost
q As (1 − ℎ! 𝑥 ) → 1, cost → ∞
q Captures intuition that larger
mistake should get larger penalties.
0 1

Slide credit: Andrew Ng

16

8
 11/9/22

Simplified Cost Function

Therefore a generalized cost function can be written as follows:

Cost ℎ' 𝑥 , 𝑦 = −𝑦log(ℎ' 𝑥 ) − ( 1 − 𝑦 log(1 − ℎ' 𝑥 ) …… (4.1)

Slide credit: Andrew Ng

17

Gradient descent

Want :
Repeat

(simultaneously update all 𝜃! )

%
𝜕 " (")
Where, 𝐽(𝜃) = - ℎ& 𝑥 − 𝑦 (") 𝑥!
𝜕𝜃!
"#$

Slide credit: Andrew Ng

18

9
 11/9/22

Multi-class classification
Disease diagnosis: health / cold / flu / pneumonia

Object classification: desk / chair / table / bookcase

19

Multi-class classification
• For 2 classes classification:
6 89: 52<
ℎ5 𝑥 = = …… (4.2)
6789: ;52< 6789: 52<

• For C classes classification {1,2,……,C}:

89: 532<
𝑝 𝑦 = 𝑖|𝑥; 𝜃= 5 = ∑6 2 ……. (4.3)
345 89: 53 <

20

10
 11/9/22

Multi-class classification
• Split into One vs Rest:

• Train a logistic regression classifier for each class i to predict

the probability that y = i with
exp 𝜃 " 𝑥
7
ℎ! (𝑥) = 9
∑78% exp 𝜃7" 𝑥

21

Any Question??

22

11