Logistic Regression

1
 Example - Two Features
x2 L L: a + b1 * x1 + b2 * x2 = 0
O
O
3 O

O ŷ = g (a + b1 * x1 + b2 * x2 )
O
X
X
X O
O
X
X X

3
x1

Logistic Regression 2
 Explanation

L is not the line of best fit as in linear

regression.
It is a line dividing regions where a + b1* x1 + b2* x2
is < and > 0.
It is the line of best separation.
Lets call f(x) = a + b1* x1 + b2* x2
yhat = g(f(x)

Logistic Regression 3
 Explanation

the further to the right of the line you are

f(x) gets larger
g(f(x)) gets closer to 1
The further to the left of the line you are
f(x) gets larger negative
g(f(x)) gets closer to 0

Logistic Regression 4
 Solution for p Dependent Variables

Estimate of P ( y =1∣x1 .. x p)=g(f ( x))

1
Estimate of P ( y =1∣x1 .. x p)= −(a + ∑k bk x k )
(1+e )

Logistic Regression 5
 Example - Decision Boundary

[Finding a, b1 and b2 later.]

Suppose a=-3 b1=1 and b2=1
Predict 1 when
-3 + x1 + x2 >= 0
x1 + x2 >= 3

Logistic Regression 6
 Example - Decision Boundary
x2 x1 + x2 – 3 > 0
O
O
3 O

O ŷ = g (-3 + x1 + x2 )
O
X
X
X O
O
X
X X Decision boundary: x1 + x2 = 3

3
x1 + x2 – 3 < 0 x1

Logistic Regression 7
Non Linear Decision Boundaries

X X
1
X
O O X
O O
O
X O
O
X
X

Logistic Regression 8
 Non Linear Decision Boundaries

Add non linear terms to the regression

equation.
ŷ = g (a + b1*x1 + b2*x2 + b3*x12 + b4 * x22 )
Get
a=-1, b1 = 0, b2=0, b3*= 1, b4=1
Decision boundary
x12 + x22 ≥ 1
Add more complex terms (e.g. x12 * x22 )to get more
complicated decision boundaries.
Logistic Regression 9
Non Linear Decision Boundaries

X X
1
X
O O X
O O
O
X O
O
X
X

Logistic Regression 10