Business Analytics and

Operations Research
Roger R. Gung, Ph.D.
Sr. Director, Business Analytics & Operations Research
roger.gung@phoenix.edu

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 1

Generalized Linear Model (GLM)

Generalized Linear Model is a flexible generalization of ordinary linear

regression that allows for dependent/target variables that have error
distribution models other than a normal distribution

GLM generalizes linear regression by allowing the linear model to be related to
the dependent/target variable via a link function

    


Two major types of GLM:

Logistic Regression
Count Regression

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 2

Logistic Regression

Logistic Regression is a regression model where the dependent/target

variable is categorical
Binary dependent variable
Multiple category dependent variable

Cases where the dependent variable has more than two outcome categories may be
analyzed in multinomial logistic regression, or,

if the multiple categories are ordered, in ordinal logistic regression

This course covers the case of a binary dependent variablethat is, where it
can take only two values, "0" and "1", which represent outcomes such as
pass/fail, win/lose, alive/dead or healthy/sick

Link function for binary dependent variable:
Logit function
Probit function

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 3

Logistic Regression

Logit function

log    


1

1
0.9
0.8
0.7
0.6
p 0.5
0.4
0.3
0.2
0.1
0
-10 -8 -6 -4 -2 0 2 4 6 8 10

(p)

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 4

Logistic Regression

Probit function: Inverse CDF of standard normal distribution

     


1
0.9
0.8
0.7
0.6
p 0.5
0.4
0.3
0.2
0.1
0
-10 -8 -6 -4 -2 0 2 4 6 8 10

(p)

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 5

Count Regression

Count Model Mean-Variance Relationship

Model Mean Variance
Poisson  
Negative Binomial (NB1)   1
Negative Binomial (NB2)   1  
Poisson Inverse Gaussian   1  2
Negative Binomial P   1  
 
Generalized Poisson  1  

Poisson Regression

log     


6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 6

SAS Proc Logistic

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 7

SAS Proc Logistic

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 8

Variable Significance Level

P-Value: Significance level of each variables impact to model goodness of fit

Odds Ratio: Significance level of each variables impact on target variable
magnitude is measured by comparing the odds of the best level to the odds of
the worst level of the driver. It is defined as the following:
" "
  !  -
1  " #$%& '$($) 1  " *+,%& '$($)

Information Value (IV) and Weight of Evidence (WOE): Significance level of each
variables impact on target variable magnitude

6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 9

Nonlinear Regression

Nonlinear Regression is a form of regression analysis in which

response/target variable is modeled by a function which is a nonlinear
combination of independent variables

Examples of nonlinear functions include exponential functions, logarithmic
functions, trigonometric functions, power functions, Gaussian function
    
. /0 

Convert . /0 to 

    



Do not convert to log     


 because is not the random
noise for 

Similarly for the following forms:
    
log 


              
6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 10
Nonlinear Regression for Marketing

Each independent variable has a lift factor 

to describe the diminishing return
(financial benefit) to the dependent variable

It is a multiplicative/interactive model that can be converted to additive model

Marketing demand model
   5 
60  7

log     
log 



9



6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 11
Binning Nonlinear Variable

It is not easy to find the best nonlinear function for each numerical variable

Convert numerical variables into categorical variables

Apply decision tree to bin numerical variables

Each bin/level will has a linear coefficient

less population



 
 
9 
: 
;


6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 12

Binning Nonlinear Variable

Binning nonlinear variables can also be applied to GLM

The dependent variable need to be judgmentally binned to observe the
nonlinearity, because it is a probability measurement

less population




 
 
9 
: 
;


6/5/2017 Copyright 2016 University of Phoenix. All Rights Reserved. 13

Receiver Operating Characteristic

Receiver operating characteristic curve, i.e. ROC curve, is a graphical plot that
illustrates the diagnostic ability of a binary classifier system as its discrimination
threshold is varied

ROC curve is created by plotting the true positive rate (TPR) against the false
positive rate (FPR) at various threshold settings

The true-positive rate is also known as sensitivity, recall or probability of
detection in machine learning

false-positive rate is also known as the fall-out or probability of false alarm and
can be calculated as (1 specicity)

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 14

ROC Basic Concept

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 15

ROC Space

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 16

Constructing ROC Curve

In binary classification, the class prediction for each instance is made based on
a continuous random variable X, which is a score computed for the instance
(e.g. estimated probability in Logistic Regression)

Given a threshold parameter T, the instance is classified as positive if X >T,
and negative otherwise

Follows a probability density
1 if the instance actually belongs to class
positive, and
0 if otherwise.

TPR is given
C
TPR  BD
1

FPR is given
C
TPR  BD
0

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 17

Area under Curve, C-statistics

ROC curve is created by plotting the true positive rate (TPR) against the false
positive rate (FPR) at various threshold settings

The area under the curve (AUC), also known as C-statistics, is equal to the
probability that a classifier will rank a randomly chosen positive instance higher
than a randomly chosen negative one

AUC is given by
C

AUC  H I J K > J
1J
0JJ K J  "L > L 

C

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 18

Weight of Evidence

In logistic regression, we usually have categorical variables. One can choose to

use Weight of Evidence (WOE) to transform categorical variables into numerical
variables

WOE of each level of a categorical variable measures the strength of the level
for separating yes and no in response variable

Benefits of WOE transformation

Linear assumption in logistic regression naturally holds (explained in next page)
Reduces degree of freedom and helps with overfitting problem
Check of collinearity between categorical variables becomes easy

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 19

WOE and Nave Bayes

For independent variable 

 1,2, P , the WOE of the Qth level is
calculated as follows:
UV!PW X. !YT !P, % !

[ \ Q\ T.].T  
_ X  X.
RS Q  TP  TP
UV!PW ^! !YT !P, % !

[ \ Q\ T.].T  
_ X  ^!

Nave Bayes classifier is given by:

 X1a  X  1
a X  1 /
a
TP  TP
 X0a X  0
a X  0 /
a

X  1
 X  1
 X  1
` X  1
 TP  TP
X  0
 X  0
 X  0
` X  0
`
X  1

X  1
 TP  TP
X  0

X  0

c

WOE vector

Linear assumption after WOE transformation holds!

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 20

WOE and Information Value (IV)

Weight of evidence indicates the predictive power of a particular level of the

variable

Information value (IV) assesses the differentiating power of the variable which
indicates how much differences from one level to other levels in regards to the
outcome, such as pass or fail

IV is given by:
Id 
 
_ X  X.   
_ X  ^! f RS Q
e)) _

One rule of thumb regarding IV is (Siddiqi, Naeem 2006):

IV < 0.02: unpredictive
0.02 <= IV < 0.1: weak
0.1 <= IV < 0.3: medium
0.3 <= IV < 0.5: strong
IV >= 0.5: suspicious and should be checked for over-predicting

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 21

Calculation of WOE & IV

Here is an example of calculating WOE and IV

Distribution_ Distribution_
Degree Level #PSI #REG #NonREG REG NonREG WOE IV
Associate 29,553 4,224 25,329 16.4% 8.1% 0.701 0.058
Bachelor 119,216 12,820 106,396 49.8% 34.2% 0.376 0.059
Master 21,911 1,774 20,137 6.9% 6.5% 0.063 0.000
Doctorate 1,268 46 1,222 0.2% 0.4% -0.787 0.002
Missing 164,978 6,871 158,107 26.7% 50.8% -0.643 0.155
ALL 336,926 25,735 311,191 100% 100% 0.274

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 22

Conversion of WOE Coefficients

When one uses WOE-transformed variables to build logistic regression, the

corresponding coefficient estimates represent the change of log odds in relation
to the increase of one unit in WOE. However, this coefficient can not be easily
interpreted.

In order to obtain the coefficients for original variables, conversion of WOE
coefficients can be performed as follows:
Multiply the coefficient with WOE for each level
Choose one level as reference (for example Undergraduate), and subtract the
multiplied result for reference from each level
Then we get the coefficient for original variable Degree Level and interpretation is
simple. For example, log odds of REG for AA PSI is 0.318 more than that of
undergraduate PSI
Degree Level WOE Coefficient Estimate Multiplied Compared with Reference (Undergraduate)
Associate 0.701 0.687 0.318
Bachelor 0.376 0.369 --
Master 0.063 0.980 0.062 -0.307
Doctorate -0.787 -0.771 -1.140
Missing -0.643 -0.631 -0.999

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 23

Survival Analysis

Survival analysis is for analyzing the expected duration of time until one or
more events happen, such as death in biological organisms and failure in
mechanical systems

KaplanMeier survival analysis is a non-parametric statistical analysis used to
estimate the survival function from lifetime data

KaplanMeier survival analysis can take into account some types of censored
data:
Right-censoring
Left-censoring

Individual 1

Individual 2 Right censoring

Individual 3

Individual 4 Left censoring

Experiment Start Time Experiment End Time

Calendar Time

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 24

KaplanMeier Survival Analysis

Individual 1

Individual 2 Right censoring

Individual 3

Individual 4 Left censoring

Survival Time

1.0
Survival Probability

2/3
2/4
1/4
Survival probability is unknown from this time point.

Survival Time

Kaplan-Meier survival probability is often viewed as actual probability.

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. 25

Survival Modeling

Survival function is the basic model employed to describe

time-to-event phenomena.

Survival function, S(t), is the probability of an individual
surviving beyond time t.

S (t ) = Pr(T > t ) = f (t )dt = 1 F (t )
t

dS (t )
f (t ) = S(t)
dt

-f(t)

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 26
Survival Modeling
- Hazard Rate/Function

Hazard rate is the death rate at time t conditional on

survival until time t or later, that is T >= t.

Suppose that an individual has survived for a time t and the
probability that it will not survive for an additional time t.
Pr(t T < t + t | T t )
h(t ) = lim t 0
t
= f (t ) / S (t )
d log[ S (t )]
=
dt
t
H (t ) = h(t )dt = log[S (t )]
0

S (t ) = e H ( t )
6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 27
Survival Modeling
- Competing Risks

Assume there are k competing risks (death factors).

Pr(t T < t + t , d = i | T t )
hi (t ) = lim t 0
t
k
hT (t ) = hi (t )
i =1

t
H T (t ) = hT (t )dt = log[S (t )]
0

S (t ) = e H T ( t )

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 28
Survival Modeling
- Parametric Regression

Defined hazard function a parametric regression function.

\  \ . 60 &g0

Parametric Regression:

log[h(t )] = log[h0 (t )] + i (t ) X i
= 0 (t ) + i (t ) X i

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 29
Survival Modeling
- Semi-parametric Regression

Cox (1972) defined proportional Hazard Rate:

\|a \ . 60 g0 60 g0 g0 

  .
\|a  \ .
60 g0

where a is the referenced attribute vector

From which, we fit parameters/coefficients for regressors

using maximum likelihood estimator, without worrying
about baseline hazard rate h0(t).

After getting coefficients, fit baseline hazard rate h0(t).

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 30
Partial Likelihood Estimator for Cox
Proportional Hazard Survival Model

In a sample size n consisting of (Tj, Xj), j = 1, 2, , n, we assume the

censoring time is non-informative in that, given Xj, the event and
censoring time for the jth individual are independent.

Let t1 < t2 < < tD denote the order of event times and X(k)i be the ith
covariates associated with the individual whose death time is tk.

Define the risk set at time tk, R(tk), as the set of all individuals who are
still under study at the time just prior to tk.

The partial likelihood based on the hazard rate is expressed by
r m
. 0no 60 g l 0
j k 5 m
0no 60 gp0
sc _q&l  .
r t r
m
log j k   
L s
 log . 0no 60 gp0

sc
c sc _q&l 

6/5/2017 Copyright 2010 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 31
Life-Span Survival Modeling
- Fitted Functions with 2 Competing Risks

Students Attributes:
Associates with High School Diploma College = School of Business
(transfer credits >= 24, FPA from Aug2009 to Aug2010) Employer Billing = No
1 Gender = Male
0.9 Marital Status = Single
0.8 Managed Own Funds = No
Graduation
0.7
Region = Northeast
Survival Probability

Withdraw Age = 35
0.6
AGI = 30k
0.5 Combined

0.4
0.3
0.2

0.1
0
101
106
111
116
121
126
131
136
141
146
151
156
161
1
6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96

Week

6/5/2017 Copyright 2012 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 32
Life-Span Survival Modeling
- Fitted Function vs Actual

Students Attributes:
Associates with High School Diploma College = School of Business
(transfer credits >=24 and FPA from Aug 2009 to Aug2010) Employer Billing = No
1.0 Gender = Male
0.9 Marital Status = Single
0.8 Managed Own Funds = No
Region = Northeast
0.7
Survival Probability

Age = any
0.6
AGI = any
0.5
0.4
Fitted
0.3
Actual
0.2
0.1
0.0
1
6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
101
106
111
116
121
126
131
136
141
146
151
156
161
Week

6/5/2017 Copyright 2012 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 33
Life-Span Survival Modeling
- Expected LS vs Actual LS

Students Attributes:
Associates with High School Diploma College = School of Business
(transfer credits >=24 and FPA from Aug 2009 to Aug2010) Employer Billing = No
Gender = Male
90 Marital Status = Single
80 Managed Own Funds = No
70 Region = Northeast
Actual LS (weeks)

60 Age = 35
50 AGI = 30k
40
30
20
10
0
20-25

25-30

30-35

35-40

40-45

45-50

50-55

55-60

60-65

65-70

70-75

75-80
Expected LS (weeks)

6/5/2017 Copyright 2012 University of Phoenix. All Rights Reserved. Business Analytics & Optimization 34