PREDICTIVE MODELING USING

SEGMENTATION

Nissan Levin NISSAN LEVIN is an independent

consultant and a partner in the
Jacob Zahavi software company Q_WARE Ltd.
in Israel.
f
JACOB ZAHAVI is Professor of
Management at the Faculty of
Management at Tel Aviv University
and the Alberto Vitale Visiting
Professor of e-commerce at the
Wharton School.

ABSTRACT DRS. ZAHAVI AND LEVIN are the

masterminds behind the
We discuss the use of segmentation as a predictive model for
development of AMOS, a
supporting targeting decisions in database marketing. We compare customized predictive modeling
the performance of judgmentally based RFM and FRAC methods to system for the Franklin Mint in
Philadelphia, and GainSmarts, a
automatic tree classifiers involving the well-known CHAID
general purpose data mining
algorithm, a variation of the AID algorithm, and a newly developed system that is the two-time winner
method based on genetic algorithm (GA). We use the logistic of the KDD-CUP competition for
the best data mining tools (1997
regression model as a benchmark for the comparative analysis. The
and 1998) sponsored by the
results indicate that automatic segmentation methods may very well American Association for Artificial
substitute the judgmentally based segmentation methods for Intelligence.

response analysis, and come only short of the logistic regression

results. The implications of the results for decision making are also
discussed.

© 2001 John Wiley & Sons, Inc. and

Direct Marketing Educational Foundation, Inc.

JOURNAL OF INTERACTIVE MARKETING

VOLUME 15 / NUMBER 2 / SPRING 2001

2
 PREDICTIVE MODELING
1. INTRODUCTION
Segmentation is key to marketing. Since cus-
tomers are not homogeneous and differ with
respect to one another in their preferences,
wants, and needs, the idea is to partition the
market into groups, or segments, of “like” peo-
ple, with similar needs and characteristics,
which are likely to exhibit similar purchasing
behavior. Then, one may offer to each segment
the products/services which are of most interest
to the members of the segment. Ideally, the
segmentation should reflect customers’ attitude
toward the product/service involved. Since this
is often not known in advance, the best proxy is
to use data that reflect customers’ purchasing
habits and behavior. Weinstein discusses several
dimensions to segmentation (1994, p. 4):
● Geography— classifying markets on the ba-
sis of geographical considerations.
●
Socioeconomic—segmentation is based on
factors reflecting customers’ socioeco-
nomic status, such as income level, educa-
tion.
● Psychographic— differentiating markets by
appealing to customers’ needs, personality
traits and lifestyle.
● Product usage—partitioning the market
based on consumption level of various us-
ers group.
● Benefits—splitting the market based upon
the benefit obtained from a product/ser-
vice, such as price, service, special features
and factors.
Hence, the segmentation process is informa-
tion based: the more information is available,
the more refined and focused are the resulting
segments.
Perhaps more than in any other marketing
channel, segmentation is especially powerful in
database marketing (DBM) where one can use
the already available wealth of information on
customers’ purchase history and demographic
and lifestyle characteristics to partition the cus-
tomer list to segments. The segmentation pro-
cess is used to distinguish between customers
and non-customers, where “customers” here are
JOURNAL OF INTERACTIVE MARKETING
extended to include buyers, payers, loyal cus-
tomers, etc., and to understand their composi-
tion and characteristics—Who they are? What
do they look like? What are their attributes?
Where do they reside? This analysis supports a
whole array of decisions, ranging from targeting
decisions to determining efficient and cost ef-
fective marketing strategies, even evaluating
market competition.
Several studies have been conducted so far
on the use of segmentation methods for sup-
porting targeting decisions. Haughton and Ou-
labi (1997) compare the performance of re-
sponse models built with CART and CHAID on
a case study that contains some 10,000 cases,
with about 30 explanatory variables and about
the same proportion of responders and nonre-
sponders. Bult and Wansbeek (1995) devise a
profit-maximization approach to selecting cus-
tomers for promotion, comparing the perfor-
mance of CHAID against several “parametric”
models (e.g., logistic regression) using a sample
of about 14,000 households with only six ex-
planatory variables. Novak, de Leeuw, and
MacEvoy (1992) devise a “richness” curve for
evaluating segmentation results, defined as the
running average of the proportion of individu-
als in a segment which are “consumers,” where
segments are added in decreasing rank order of
their response. Morwitz and Schmittlein (1992)
investigate whether the use of segmentation can
improve the accuracy of sales forecasts based on
stated purchase intent involving CART, dis-
criminant analysis, and K-means clustering algo-
rithm. Other attempts have been made to im-
prove the targeting decisions with segmentation
by using prior information. Of these we men-
tion the parametric approach of Morwitz and
Schmittlein (1998) and the nonparametric ap-
proach of Levin and Zahavi (1996).
In this paper we discuss the use of segmenta-
tion as a predictive model for response model-
ing in database marketing. We compare the
performance of judgmentally based methods to
several automatic tree-structured segmentation
methods (decision trees); two are extensions of
the popular AID and CHAID trees, and the
third is a new decision tree based on Genetic
Algorithm (GA). To assess how good the pre-
● VOLUME 15 / NUMBER 2 / SPRING 2001
3
 JOURNAL OF INTERACTIVE MARKETING
diction power of the segmentation-based mod-
els is, we compare them against the results of a
logistic regression model, which is undoubtedly
one of the most advanced response models in
database marketing. Logistic regression is
widely discussed in the literature and will not be
reviewed here (see, e.g., Ben-Akiva & Lerman,
1987; Long, 1997). The development and eval-
uation of the decision trees were conducted by
the authors’ own computer programs. The
study itself was conducted on real data from the
collectible industry.
To compare the merit of our methods versus
the other tree-based methods for response mod-
eling mentioned above, one needs to run all
methods on the same set of data, which is not
feasible. Also, since tree classifiers are heuristic
methods, the decision tree results are only as
good as the algorithm used to create the trees.
Hence, all results in this paper reflect the per-
formance of our computer algorithms, which
may not extend to other algorithms.
Yet, this paper offers several contributions
over previous work in this area. On the theoret-
ical front, we extend the decision tree algo-
rithms in several directions, as discussed in the
next section. We also offer a unified and gener-
alized framework to formulate decision trees for
segmenting an audience.
On the practical front, this paper provides
hard empirical evidence of the relative merits of
various segmentation methods, focusing on sev-
eral issues:
● How well automatic tree classifiers are ca-
pable of discriminating between buying
and nonbuying segments.
● How well automatic segmentation per-
forms as compared to manually based seg-
mentation.
● How the various automatic tree classifiers
compare against logistic regression results.
● What practical implications one needs to
look into when using automatic segmenta-
tion.
For confidentiality reasons, all results are pre-
sented in percentage terms. We also discuss the
implications of the results for decision making.
JOURNAL OF INTERACTIVE MARKETING
2. SEGMENTATION METHODS
2.1 Judgmentally Based Methods
Judgmentally based or “manual” segmentation
methods are still most commonly used in DBM
to partition a customers’ list into “homoge-
neous” segments. Typical segmentation criteria
include previous purchase behavior, demo-
graphics, geographics, and psychographics. Pre-
vious purchase behavior is often considered to
be the most powerful criterion in predicting the
likelihood of future response. This criterion is
operationalized for the segmentation process by
means of Recency, Frequency, and Monetary
(RFM) variables (Shepard, 1995). Recency cor-
responds to the number of weeks (or months)
since the most recent purchase, or the number
of mailings since last purchase; Frequency to
the number of previous purchases or the pro-
portion of mailings to which the customer re-
sponded; and Monetary to the total amount of
money spent on all purchases (or purchases
within a product category), or the average
amount of money per purchase. The general
convention in the DBM industry is that the
more recently the customer has placed the last
order, the more items he/she bought from the
company in the past, and the more money he/
she spent on the company’s products, the
higher is his/her likelihood of purchasing the
next offering and the better target he/she is.
This simple rule allows one to arrange the seg-
ments in decreasing likelihood of purchase.
The more sophisticated manual methods also
make use of product/attribute proximity con-
siderations in segmenting a file. By and large,
the more similar the products bought in the
past are to the current product offering, or the
more related are the attributes (e.g., themes),
the higher the likelihood of purchase. For ex-
ample, in a book club application, customers
are segmented based upon the proximity of the
theme/content of the current book to those of
previously purchased books. Say, the currently
promoted book is The Art History of Florence, then
book club members who previously bought Ital-
ian art books are the most likely candidates to
buy the new book, and are therefore placed at
the top of the segmentation list, followed by
● VOLUME 15 / NUMBER 2 / SPRING 2001
4
 PREDICTIVE MODELING
people who purchased general art books, fol-
lowed by people who purchased geographical
books, and so on. In cases where males and
females may react differently to the product
offering, gender may also be used to partition
customers into groups. Generally, the list is first
partitioned by product/attribute type, then by
RFM, and then by gender (i.e., the segmenta-
tion process is hierarchical). This segmentation
scheme is also known as FRAC—Frequency, Re-
cency, Amount (of money), and Category (of
product) (Kestnbaum, 1998).
Manually based RFM and FRAC methods are
subject to judgmental and subjective consider-
ations. Also, the basic assumption behind the
RFM method may not always hold. For example,
in durable products, such as cars or refrigera-
tors, recency may work in a reverse way—the
longer the time since last purchase, the higher
the likelihood of purchase. Finally, to meet seg-
ment size constraints, it may be necessary to run
the RFM/FRAC iteratively, each time combin-
ing small segments and splitting up large seg-
ments, until a satisfactory solution is obtained.
This may increase computation time signifi-
cantly.
2.2 Decision Trees
Several “automatic” methods have been devised
in the literature to take away the judgmental
and subjective considerations inherent in the
manual segmentation process. By and large,
these methods map data items (customers, in
our case) into one of several predefined classes.
In the most simple case, the purpose is to seg-
ment customers into one of two classes, based
on some type of a binary response, such as
buy/no buy, loyal/nonloyal, pay/no-pay, etc.
Thus, tree classifiers are choice based. Without
loss of generality we refer to the choice variable
throughout this paper as purchase/no-pur-
chase, thus classifying the customers into seg-
ments of “buyers” and “nonbuyers.”
Several automatic tree classifiers were dis-
cussed in the literature, among them AID (Au-
tomatic Interaction Detection; Sonquist, Baker,
& Morgan, 1971); CHAID (Chi square AID;
Kass, 1983); CART (Classification and Regres-
sion Trees; Breiman, Friedman, Olshen, &
JOURNAL OF INTERACTIVE MARKETING
Stone, 1984); ID3 (Quinlan, 1986); C4.5 (Quin-
lan, 1993); and others. For a comprehensive
survey of automatic construction of decision
trees from data see Murthy (1998).
Basically, all automatic tree classifiers share
the same structure. Starting from a “root” node
(the whole population), tree classifiers employ
a systematic approach to grow a tree into
“branches” and “leaves.” In each stage, the al-
gorithm looks for the “best” way to split a “fa-
ther” node into several “children” nodes, based
on some splitting criteria. Then, using a set of
predefined termination rules, some nodes are
declared as “undetermined” and become the
father nodes in the next stages of the tree de-
velopment process, some others are declared as
“terminal” nodes. The process proceeds in this
way until no more node is left in the tree which
is worth splitting any further. The terminal
nodes define the resulting segments. If each
node in a tree is split into two children only,
one of which is a terminal node, the tree is said
to be “hierarchical.”
Three main considerations are involved in
developing automatic trees:
1. Growing the tree
2. Determining the best split
3. Termination rules
Growing the Tree. One grows the tree by suc-
cessively partitioning nodes based on the data.
A node may be partitioned based on several
variables at a time, or even a function of vari-
ables (e.g., a linear combination of variables).
With so many variables involved, there are prac-
tically infinite number of ways to split a node.
Take as an example just a single continuous
variable. This variable alone can be partitioned
in infinite number of ways, let alone when sev-
eral variables are involved.
In addition, each node may be partitioned
into several descendants (or splits), each be-
comes a “father” node to be partitioned in the
next stage of the tree development process.
Thus, the larger the number of splits per node,
the larger the tree and the more prohibitive the
calculations.
Indeed, several methods have been applied
● VOLUME 15 / NUMBER 2 / SPRING 2001
5
 JOURNAL OF INTERACTIVE MARKETING
in practice to reduce the number of possible
partitions of a node:
I. All continuous variables are categorized
prior to the tree development into small num-
ber of ranges (“binning”). A similar procedure
applies for integer variables, which assume
many values (such as the frequency of pur-
chase).
II. Nodes are partitioned only on one variable
at a time (“univariate” algorithm).
III. The number of splits per each “father”
node is often restricted to two (“binary” trees).
IV. Splits are based on a “greedy” algorithm
in which splitting decisions are made sequen-
tially looking only on the impact of the split in
the current stage, but never beyond (i.e., there
is no “looking ahead”).
Determining the Best Split. With so many pos-
sible partitions per node, the question is what is
the best split? There is no unique answer to this
question as one may use variety of splitting cri-
teria, each may result in a different “best” split.
We can classify the splitting criteria into two
“families”: node-value based criteria and parti-
tion-value based criteria.
● Node-value based criteria: seeking the split
that yields the best improvement in the
node value.
● Partition-value based criteria: seeking the
split that separates the node into groups
that are as different from each other as
possible.
We discuss the splitting criteria at more
length in Appendix A.
Termination Rules. Theoretically, one can
grow a tree indefinitely, until all terminal nodes
contain very few customers, as low as one cus-
tomer per segment. The resulting tree in this
case is unbounded and unintelligible, having
the effect of “can’t see the forest because of too
many trees.” It misses the whole point of tree
classifiers, whose purpose is to divide the popu-
lation into buckets of “like” people, where each
bucket contains a meaningful number of peo-
ple for statistical significance. In addition, the
JOURNAL OF INTERACTIVE MARKETING
larger the tree, the larger the risk of overfitting.
Hence it is necessary to control the size of a tree
by means of termination rules that determine
when to stop growing the tree. These termina-
tion rules should be set to ensure statistical
validity of the results and avoid overfitting.
2.3 Extended Tree Classifiers
We conclude this section by discussing the three
tree classifiers that we use in our paper: a vari-
ation of AID, which we refer to as Standard Tree
Algorithm (STA); the commonly used CHAID,
which was extended to consider entropy crite-
rion; and a new tree classifier based on Genetic
Algorithm (GA). Basically, AID, STA and
CHAID are combinatorial algorithms in the
sense that they go over all possible combina-
tions of variables to partition a node. Conse-
quently, these algorithms are computationally
intensive and therefore, as discussed above, are
limited to splits based on one variable, or at best
two variables, at a time. In contrast, GA is a
non-combinatorial algorithm in the sense that
the candidate solutions (splits) are generated
by a random, yet systematic, search method.
This opens up the possibility to consider parti-
tions based on more than two variables at a
time, hopefully yielding more “homogeneous”
segments. Practically, this renders the GA algo-
rithm “non-greedy” by making it possible to
look ahead beyond the current stage when split-
ting a node. However, because of computa-
tional constraints, we have limited the number
of simultaneous variables to split a node to only
three and four predictors at a time.
These algorithms are further described in
Appendix B.
3. A CASE STUDY
We use a real case study and actual data to
demonstrate and evaluate the performance of
decision trees as compared to manually based
trees. The case study involves a solo mailing of a
collectible item that was live-tested in the mar-
ket and then rolled out. The data for the anal-
ysis consists of the test audience with appended
orders, containing 59,970 customers, which we
randomly split into two mutually exclusive sam-
● VOLUME 15 / NUMBER 2 / SPRING 2001
6
 PREDICTIVE MODELING
ples—a calibration (training) sample, consist-
ing of 60% of the observations, to build the
model (tree), and a holdout sample, containing
the rest of the customers, to validate the model.
As alluded to earlier, only binary predictors
may take part in the partitioning process.
Hence, all continuous and multivalued integer
variables were categorized, prior to invoking the
tree algorithm, into ranges, each is represented
by a binary variable assuming the value of 1 if
the variable falls in the interval, 0 otherwise.
This process is also referred to as “binning.”
Depending upon the tree, the resulting catego-
ries may be either overlapping (i.e., X ⱕ ai, i
⫽ 1, 2, . . . , where X is a predictor, ai a given
breakpoint) or non-overlapping (i.e., ai-1 ⱕ X
⬍ ai, i ⫽ 1, 2, . . . ).
The trees were evaluated using various good-
ness-of-fit criteria that express how well the seg-
mentation process is capable of discriminating
between buyers and nonbuyers. A common
measure is in terms of the percentage of buyers
“captured” per the percentage of audience
mailed, the higher the percentage of buyers,
the “better” the model. For example, a segmen-
tation scheme that captures 80% of the buyers
for 50% of the audience is better than a seg-
mentation scheme that captures only 70% of
the buyers for the same audience.
Below we discuss several considerations in
setting up the study.
Feature Selection
In DBM applications, the number of potential
predictors could be very large, often in the or-
der of magnitude of several hundreds, even
more, predictors. Thus, the crux of the model-
building process is to pick the handful of pre-
dictors that “best” explain, from a statistical
point of view, the customers’ choice decision.
This problem, referred to as the feature selec-
tion (or specification) problem, is a tough com-
binatorial problem and is definitely the most
complex issue in building large-scale multivari-
ate models.
Decision trees possess an advantage over sta-
tistical regression models in that they have an
inherent mechanism to pick the predictors af-
fecting the terminal segments. They do it by
JOURNAL OF INTERACTIVE MARKETING
going over all possible combinations to grow a
tree (subject to the computational constraints
discussed in the previous section) and selecting
the best split for each node using one of many
splitting criteria (discussed in Appendix A).
In contrast to decision trees, logistic regres-
sion models do not enjoy this benefit, and one
needs to invoke a specification procedure to
select the most influential predictors to include
in the final model. Hence, using logistic regres-
sion models with multiple potential predictors
is not easy and definitely not as straightforward
as building decision trees. In our case, we use a
rule-based expert system to weed out the “bad”
from the “good” predictors. These rules were
calibrated, based on statistical theory and prac-
tice, using an extensive experimentation pro-
cess.
The Predictors Set
The mix of predictors is a major factor affecting
the resulting tree structure; the larger the num-
ber of potential predictors to split a tree, the
larger is the number of segments and the
smaller is the size of each terminal segment. To
determine the impact of the mix and the num-
ber of predictors on the performance of the
tree classifiers, we have experimented with sev-
eral sets of predictors. The basic set, the affinity
set, contains the basic attributes of collectible
items; the second set also includes recency pre-
dictors; the third set is augmented to include
frequency measures. Finally, the last set con-
tains all of the predictors available in the data
set, several hundreds in all. More details about
the participating predictors are found in Ap-
pendix C.
Min/Max Segment Size
Size constraints are most crucial in segmenta-
tion analysis. To minimize the error incurred in
case wrong decisions are made (e.g., because of
sampling errors), segments are required to be
“not-too-small” and “not-too-big.” If a segment is
too small, the probability of making an error
increases due to the lack of statistical signifi-
cance. If the segment is too big, then if a “good”
segment is somehow eliminated from the mail-
ing (Type I error)—large foregone profits are
● VOLUME 15 / NUMBER 2 / SPRING 2001
7
 JOURNAL OF INTERACTIVE MARKETING
incurred; and if a “bad” segment makes it to the
mailing (Type II error)— large out-of-pocket
costs are incurred.
Consequently, we have built a mechanism in
all our automatic tree algorithms to account for
minimum and maximum constraints on the re-
sulting segment size. In our study, we used two
sets of min/max constraints on the segment
size, 150/3,000 and 300/6,000, respectively.
Splitting Criteria
Finally, as discussed in Appendix A, one may
define a variety of splitting criteria which be-
long to the node-value and partition-value fam-
ilies. We have used four different criteria in our
study all of which seek to maximize a statistical
measure Y as follows.
Criterion 1 (CHAID). Y is the statistic for the
chi-square test of independence:
Y⫽ 冘
splits
共Observed ⫺ Expected兲2
Expected
Clearly, the larger the value of Y, the larger is
the difference between the response rates of the
resulting child nodes, and the “better” the split.
This statistic is distributed as chi-square with (k
⫺ 1) degrees of freedom, where k is the number
splits for the node. Then, if the resulting P_
value is less than or equal to the level of signif-
icance, we conclude that Y is big “enough” and
that the resulting split is “good.” Since CHAID
uses a sequence of tests (each possible split
constitutes a test), an adjusted P_ value measure
using the Bonferroni multiplier, is often used to
determine the “best” split.
Criterion 2 (CHAID). Y is the total entropy
associated with a given partition (into M splits).
It is a measure of the information content of the
split, the larger the entropy the better.
Criterion 3 (STA). Y is the number of stan-
dard deviations that the response rate (RR) of
the smaller child node (the one with the fewer
number of customers) is away from the overall
response rate of the training audience (TRR).
JOURNAL OF INTERACTIVE MARKETING
Large values of Y (e.g., Y ⱖ 2) mean that the
true (but unknown) response rate of the result-
ing segment is significantly different from the
TRR, indicating a “good” split.
Criterion 4 (STA, GA). Y is the larger re-
sponse rate of the two children nodes (in a
binary split). This criterion seeks the split that
maximize the difference in the response rates of
the two descendant nodes.
All these criteria are further discussed in Ap-
pendix A.
4. RESULTS AND ANALYSIS
The combination of several tree classifiers, pre-
dictor sets, and splitting criteria give rise to
numerous trees. To simplify the presentation,
we provide only selective results in this paper.
We evaluate all trees based on goodness-of-fit
criteria. As a reference point, we compare the
automatic segmentation and the manually
based segmentation to logistic regression.
To allow for a “fair” comparison between the
models, each model was optimized to yield the
best results: the manually based segmentation
by using domain experts to determine the
FRAC segmentation criteria; the automatic de-
cision trees by finding the best tree for the given
splitting criterion and constraints; and the lo-
gistic regression by finding the “best” set of
predictors that explain customer choice.
Goodness-of-Fit Results
In this section we analyze the performance of
each method on its own, using goodness-of-fit
measures that exhibit how well a model is capa-
ble of discriminating between buyers and non-
buyers. Goodness-of-fit results are typically pre-
sented by means of gains tables in descending
order of the predicted response. In the segmen-
tation-based methods, the predicted response
rate of a rollout segment is given by the corre-
sponding response rate of the segment in the
calibration (training) sample. In logistic regres-
sion, the response rates are estimated directly
by the model. Several measures can be used to
assess the performance of a response model:
● VOLUME 15 / NUMBER 2 / SPRING 2001
8
 PREDICTIVE MODELING
● The behavior of the actual response rates
(the ratio of the number of buyers “cap-
tured” to the size of the audience mailed)
across segments, which in a “good” model
should exhibit a nicely declining pattern.
● The difference between the actual re-
sponse rates of the top and the bottom
segments, the larger the difference, the
better the fit.
All the results below are presented for the
holdout sample. Table 1 presents the gains ta-
ble for RFM-based segmentation; Table 2, for
FRAC-based segmentation. Out of the many
tree classifiers that we analyzed, we present one
gains tables for STA, with 1-variable split, split-
ting criterion 4, a predictor set that contains the
affinity and recency attributes and min/max
constraints on the resulting segment size of 150
and 3,000 respectively (Table 3). Finally, Table
4 exhibits the logistic regression results by de-
ciles using the entire set of predictors.
Observing Table 1 (RFM segmentation),
other than the top two segments, the response
rates across segments in the list are pretty flat
and somewhat fluctuating— both are indicative
of relatively poor fit.
By comparison, in the FRAC segmentation
(Table 2), the top segments perform signifi-
cantly better than the bottom segments, with
the top segment yielding a response rate of
12.99% versus an average response rate of only
.69% for the entire holdout audience.
Interestingly, the automatic segmentation
methods (Table 3) are not lagging behind in
terms of discriminating between the buying and
the nonbuying segments, with the top segment
outperforming the bottom segments by a wide
margin.
Finally, the logistic regression model (Table
4) exhibits excellent performance, with the top
decile responding more than six times better
than the bottom decile.
Tree Performance
To evaluate and compare the automatic seg-
mentation to the judgmentally based segmenta-
tion and the logistic regression model, we look
on the percentage of buyers captured at several
JOURNAL OF INTERACTIVE MARKETING
representative audiences. The reference point
consists of the top 30% of the customers in the
list of segments, arranged in descending order
of the response rate of the segments in the
calibration sample. Note that in a tree analysis,
the response probability of a customer is deter-
mined by the response rate of his/her peers,
that is, the response rate of the segment that the
customer belongs to. Since segments’ sizes are
discrete, we use interpolation to exhibit the
performance results at exactly 30% of the audi-
ence. Of course, no interpolation is required
for logistic regression, since here the response
probability is calculated individually for each
customer in the list. As additional reference
points, we also present the performance results
for the top 10% and 50% of the audience.
Table 5 presents the performance results at
these audience levels for several tree classifiers,
as well as the results of RFM, FRAC, and logistic
regression. To bring all methods to an equal
footing for comparison purposes, all methods
were calibrated on the same data set using the
entire set of predictors. Comparing the results,
we conclude:
● The logistic regression model outperforms
all other models—the judgmentally based,
as well as the automatic models.
● The RFM-based models are the worst.
● The automatic tree classifiers perform ex-
tremely well, being comparable to the
FRAC-based model and getting close to the
logistic regression model.
● Increasing the minimum size constraint
usually increases the variance of the fit re-
sults across all segments of a tree. This is a
manifestation of the fact that larger seg-
ments are less “homogeneous” and thus
exhibit larger variation. Indeed, smaller
segments are more stable, but increase the
risk of overfitting. So one needs to trade off
variance against segment size to find the
most suitable one for the occasion. Unfor-
tunately, there is no optimal way to find the
“best” segment size other than by experi-
mentation.
We note that adding more predictors to the
● VOLUME 15 / NUMBER 2 / SPRING 2001
9
 JOURNAL OF INTERACTIVE MARKETING

T A B L E 1
Gains Table for RFM-Based Segmentation
Segments with at Least 100 Customers in the Holdout Sample Sorted by Descending Response Rates of the Segments in
the Calibration Sample.
CLB HLD CUM CLB CUM HLD CUM CLB CUM CLB CUM HLD CUM HLD %CLB AUD/
SEG RR % RR % RR % RR % BUY % AUD % BUY % AUD % %HLD AUD

144 2.38 2.62 2.38 2.62 29.22 7.47 27.88 7.68 0.98
244 1.90 2.38 2.27 2.47 36.07 9.66 34.55 9.61 1.14
322 1.24 0.58 2.21 2.34 37.44 10.33 35.15 10.32 0.94
434 1.10 0.86 2.11 2.21 39.27 11.34 36.36 11.29 1.04
423 0.96 0.00 1.98 1.99 41.55 12.75 36.36 12.60 1.05
223 0.90 1.02 1.91 1.93 42.92 13.67 37.58 13.42 1.12
143 0.90 0.94 1.83 1.84 44.75 14.91 39.39 14.75 0.93
333 0.87 1.43 1.72 1.79 47.49 16.83 43.64 16.78 0.94
433 0.86 1.02 1.63 1.71 50.23 18.76 46.67 18.82 0.95
134 0.84 0.62 1.58 1.63 52.05 20.09 47.88 20.16 0.99
344 0.78 1.23 1.42 1.55 58.45 25.08 56.97 25.23 0.98
334 0.75 0.56 1.36 1.47 61.19 27.29 58.79 27.47 0.99
131 0.72 0.00 1.35 1.44 62.10 28.06 58.79 28.13 1.16
133 0.71 0.88 1.27 1.36 66.67 31.97 64.24 32.41 0.91
123 0.67 1.02 1.24 1.35 68.49 33.63 66.67 34.04 1.01
233 0.62 1.82 1.22 1.36 69.41 34.52 69.09 34.96 0.97
122 0.58 0.38 1.17 1.27 72.60 37.88 70.91 38.29 1.01
412 0.58 0.67 1.14 1.25 74.43 39.81 72.73 40.17 1.03
323 0.57 0.33 1.11 1.19 76.71 42.23 73.94 42.69 0.96
132 0.46 0.37 1.07 1.15 78.54 44.65 75.15 44.95 1.07
213 0.45 0.00 1.06 1.14 79.00 45.27 75.15 45.54 1.03
234 0.44 0.00 1.05 1.12 79.45 45.90 75.15 46.15 1.05
312 0.39 1.02 1.01 1.11 81.28 48.76 79.39 49.00 1.00
212 0.38 0.24 0.95 1.03 84.47 53.90 81.21 54.28 0.97
211 0.34 0.27 0.86 0.92 89.50 62.98 84.85 63.44 0.99
411 0.28 0.37 0.81 0.87 92.69 69.81 88.48 70.19 1.01
112 0.28 0.40 0.79 0.85 93.61 71.81 89.70 72.29 0.95
444 0.27 0.42 0.76 0.83 95.43 75.93 92.12 76.26 1.04
311 0.26 0.27 0.74 0.81 96.80 79.11 93.33 79.31 1.04
432 0.26 0.85 0.74 0.81 97.26 80.18 94.55 80.29 1.09
413 0.23 0.00 0.72 0.79 98.17 82.63 94.55 82.54 1.09
111 0.20 0.46 0.71 0.78 99.09 85.40 96.36 85.28 1.01
121 0.17 0.00 0.70 0.76 99.54 87.02 96.36 86.79 1.08
313 0.07 0.21 0.67 0.74 100.00 90.94 97.58 90.73 0.99
All 0.61 0.69 100.00 100.00 100.00 100.00 0.98

CLB ⫽ Calibration sample

HLD ⫽ Holdout sample
CUM ⫽ Cumulative
SEG ⫽ RFM segment number:
1st digit-Recency: 1-most recent 3 4-least recent
2nd digit-Frequency: 1-few purchases 3 4-many purchases
3rd digit Monetary: 1-least spending 3 4 most spending

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

10
 PREDICTIVE MODELING

T A B L E 2
Gains Table for FRAC-Based Segmentation
Segments with at Least 100 Customers in the Holdout Sample Sorted by Descending Response Rates of the Segments in
the Calibration Sample.
CLB HLD CUM CLB CUM HLD CUM CLB CUM CLB CUM HLD CUM HLD %CLB AUD/
SEG RR % RR % RR % RR % BUY % AUD % BUY % AUD % %HLD AUD

1 12.36 12.99 12.36 12.99 14.61 0.72 13.94 0.74 0.98

2 6.42 6.53 8.92 8.67 31.96 2.18 26.06 2.07 1.10
3 5.07 4.92 7.59 7.37 41.55 3.33 33.94 3.17 1.05
4 3.73 3.79 6.75 6.29 47.95 4.32 38.79 4.24 0.93
5 3.51 3.41 6.05 5.64 54.79 5.51 44.85 5.47 0.97
6 2.33 4.42 5.33 5.39 59.82 6.83 53.94 6.88 0.93
7 2.33 1.53 5.01 4.98 63.01 7.66 55.76 7.70 1.02
8 1.78 3.02 4.51 4.60 67.12 9.07 61.82 9.24 0.90
9 1.52 2.22 4.24 4.35 69.41 9.95 64.24 10.15 0.97
10 1.08 0.90 3.84 3.91 72.15 11.43 66.67 11.72 0.92
11 0.86 1.87 3.54 3.70 73.97 12.72 70.30 13.06 0.96
12 0.84 0.00 3.28 3.36 75.80 14.05 70.30 14.38 1.01
13 0.80 0.00 3.06 3.08 77.63 15.43 70.30 15.71 1.04
14 0.46 0.70 2.79 2.83 79.00 17.25 72.12 17.50 1.02
15 0.45 0.22 2.38 2.36 81.74 20.93 73.33 21.33 0.96
16 0.44 0.34 2.27 2.25 82.65 22.19 73.94 22.57 1.02
17 0.42 0.00 2.16 2.13 83.56 23.52 73.94 23.82 1.06
18 0.36 0.27 2.05 2.02 84.47 25.06 74.55 25.34 1.02
19 0.36 0.27 1.87 1.83 86.30 28.15 75.76 28.46 0.99
20 0.35 0.78 1.71 1.73 88.13 31.33 79.39 31.65 1.00
21 0.26 0.16 1.43 1.42 91.32 38.95 81.21 39.44 0.98
22 0.19 0.37 1.30 1.31 92.69 43.43 83.64 43.91 1.00
23 0.19 0.15 1.23 1.24 93.61 46.43 84.24 46.74 1.06
24 0.16 0.48 1.19 1.21 94.06 48.20 85.45 48.47 1.02
25 0.13 0.13 1.06 1.08 95.43 54.73 86.67 54.96 1.01
26 0.12 0.25 0.99 1.02 96.35 59.37 88.48 59.96 0.93
27 0.11 0.00 0.95 0.97 96.80 61.95 88.48 62.67 0.95
28 0.09 0.05 0.84 0.86 98.17 71.27 89.09 71.65 1.04
29 0.09 0.20 0.78 0.80 99.09 77.60 90.91 77.76 1.04
30 0.07 0.65 0.74 0.80 99.54 81.39 94.55 81.60 0.99
31 0.07 0.12 0.71 0.77 100.00 85.20 95.15 85.20 1.06
All 0.61 0.69 100.00 100.00 100.00 100.00 1.00

CLB ⫽ Calibration sample

HLD ⫽ Holdout sample
CUM ⫽ Cumulative
SEG ⫽ Sequential segment number

process does not necessarily improve the frequency variables adding very little to im-
model fit. It turns out that in our case the proving the fit. This phenomenon may be
affinity consideration possess the most predic- typical in the collectible industry, where a
tion power, with the additional recency and customer either likes a given range of prod-

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

11
 JOURNAL OF INTERACTIVE MARKETING

T A B L E 3
Gains Table for STA, 1-Variable Split, Affinity and Recency Predictors, Splitting Criterion 4, and Min/Max Constraint
150/3000
Segments with at Least 25 Customers in the Holdout Sample Sorted by Descending Response Rates of the Segments in the
Calibration Sample.
CLB HLD CUM CLB CUM HLD CUM CLB CUM CLB CUM HLD CUM HLD %CLB AUD/
SEG RR % RR % RR % RR % BUY % AUD % BUY % AUD % %HLD AUD

9 8.12 10.08 8.12 10.08 19.18 1.44 21.82 1.49 0.97

1 5.65 1.85 7.36 7.51 25.11 2.08 23.64 2.16 0.95
8 4.55 6.50 6.42 7.19 32.88 3.12 33.33 3.19 1.01
10 4.05 4.20 6.03 6.72 36.99 3.73 36.97 3.79 1.03
2 4.01 3.67 5.64 6.13 42.92 4.63 41.82 4.69 0.99
15 3.49 3.46 5.25 5.67 48.86 5.67 46.67 5.66 1.07
13 3.37 3.85 4.99 5.43 53.88 6.57 51.52 6.52 1.04
4 3.31 4.67 4.89 5.38 56.16 6.99 54.55 6.97 0.94
24 3.11 0.93 4.78 5.11 58.45 7.44 55.15 7.42 0.99
16 2.41 0.00 4.64 4.85 60.27 7.90 55.15 7.83 1.13
18 1.84 1.81 4.27 4.46 63.93 9.11 58.18 8.98 1.05
22 1.75 2.87 4.11 4.32 65.75 9.74 61.82 9.85 0.73
12 1.51 1.14 3.92 4.10 67.58 10.48 63.03 10.58 1.00
14 1.39 2.97 3.74 4.01 69.41 11.28 66.67 11.43 0.95
34 1.21 0.00 3.60 3.80 70.78 11.97 66.67 12.06 1.09
3 1.09 0.39 3.40 3.52 72.60 12.99 67.27 13.13 0.95
20 1.05 0.00 3.31 3.40 73.52 13.52 67.27 13.61 1.10
23 0.96 1.00 3.13 3.20 75.34 14.67 69.09 14.86 0.92
30 0.96 0.96 3.00 3.07 76.71 15.54 70.30 15.73 1.00
7 0.90 0.90 2.89 2.95 78.08 16.47 71.52 16.66 1.00
5 0.75 0.61 2.79 2.86 79.00 17.21 72.12 17.34 1.09
11 0.70 0.00 2.70 2.74 79.91 18.01 72.12 18.12 1.02
32 0.68 0.47 2.61 2.63 80.82 18.83 72.73 19.01 0.92
17 0.65 0.00 2.57 2.57 81.28 19.26 72.73 19.45 0.98
19 0.63 0.40 2.48 2.46 82.19 20.14 73.33 20.49 0.85
25 0.53 0.00 2.43 2.41 82.65 20.66 73.33 20.96 1.09
60 0.49 0.78 2.38 2.37 83.11 21.23 73.94 21.50 1.05
38 0.48 0.00 2.33 2.31 83.56 21.81 73.94 21.98 1.21
40 0.47 0.00 2.28 2.25 84.02 22.39 73.94 22.58 0.97
41 0.46 0.00 2.24 2.20 84.47 22.99 73.94 23.13 1.09
48 0.46 0.00 2.19 2.14 84.93 23.60 73.94 23.72 1.03
31 0.38 0.00 2.14 2.08 85.39 24.34 73.94 24.47 0.99
26 0.37 1.04 2.08 2.05 85.84 25.10 75.15 25.27 0.94
59 0.36 0.00 2.03 1.99 86.30 25.87 75.15 25.95 1.12
47 0.32 0.65 1.87 1.87 87.67 28.48 77.58 28.53 1.01
42 0.31 0.00 1.83 1.81 88.13 29.37 77.58 29.46 0.96
39 0.30 0.46 1.78 1.77 88.58 30.28 78.18 30.37 1.00
21 0.30 1.32 1.74 1.76 89.04 31.21 80.00 31.32 0.97
37 0.30 0.85 1.69 1.73 89.50 32.15 81.21 32.30 0.96
49 0.29 0.00 1.65 1.68 89.95 33.11 81.21 33.30 0.96

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

12
 PREDICTIVE MODELING

T A B L E 3
Continued
Segments with at Least 25 Customers in the Holdout Sample Sorted by Descending Response Rates of the Segments in the
Calibration Sample.
CLB HLD CUM CLB CUM HLD CUM CLB CUM CLB CUM HLD CUM HLD %CLB AUD/
SEG RR % RR % RR % RR % BUY % AUD % BUY % AUD % %HLD AUD

50 0.28 0.00 1.61 1.63 90.41 34.10 81.21 34.18 1.12

54 0.27 0.28 1.50 1.53 91.78 37.18 82.42 37.11 1.05
64 0.27 0.68 1.47 1.50 92.24 38.22 83.64 38.34 0.84
63 0.27 0.21 1.41 1.44 93.15 40.30 84.24 40.37 1.03
33 0.24 0.19 1.34 1.37 94.06 42.62 84.85 42.60 1.04
69 0.24 0.36 1.31 1.34 94.52 43.79 85.45 43.74 1.02
6 0.23 0.33 1.28 1.32 94.89 44.99 86.06 45.00 0.96
35 0.15 0.11 1.20 1.22 95.89 48.72 86.67 48.76 0.99
53 0.13 0.19 1.11 1.14 96.80 52.88 87.88 53.09 0.96
57 0.13 0.21 1.04 1.08 97.72 57.15 89.09 56.99 1.09
65 0.13 0.29 0.92 0.97 99.54 65.67 92.73 65.48 1.00
43 0.13 0.00 0.90 0.94 100.00 67.84 92.73 67.64 1.00
All 0.61 0.69 100.00 100.00 100.00 100.00 0.99

CLB ⫽ Calibration sample

HLD ⫽ Holdout sample
CUM ⫽ Cumulative
SEG ⫽ STA segment number

ucts (e.g., dolls) or not, and may not extend the various tree classifiers to one another to find
to other industries. out which one performs the best. But this re-
Finally, it would be interesting to compare quires extensive experimentation with multiple
tree classifiers and many data sets, which was
beyond the scope of this research.
T A B L E 4
Gains Table for Logistic Regression by Decils Based on
Entire Set of Predictors
6. CONCLUSIONS
Actual In this paper we evaluated the performance of
% % Resp % Response/ Pred Resp automatic tree classifiers versus the judgmen-
Prospects Response Rate % Prospects Rate %
tally based RFM and FRAC methods and logistic
10.00 62.42 4.29 6.24 4.26
regression. The methods were evaluated based
20.00 77.58 2.67 3.88 2.57
on goodness-of-fit measures, using real data
30.00 83.03 1.90 2.77 1.83
from the collectible industry. Three automatic
40.00 86.67 1.49 2.17 1.43 tree classifiers participated in the analysis—a
50.00 89.70 1.23 1.79 1.18 modified AID tree that we term STA, an exten-
60.00 92.12 1.06 1.53 1.00 sion of the CHAID method, and a newly devel-
70.00 96.36 0.95 1.38 0.87 oped GA tree.
80.00 98.18 0.84 1.23 0.78 The evaluation process, which involves sev-
90.00 100.00 0.76 1.11 0.70 eral predictor sets, several splitting criteria and
100.00 100.00 0.69 1.00 0.63 several constraints on the minimum and maxi-
mum size of the terminal segments, shows that

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

13
 JOURNAL OF INTERACTIVE MARKETING

traditional statistical methods, no extensive

T A B L E 5
background in statistics is required to build
Summary of Performance Results Based on Entire Set
of Predictors trees (as the feature selection process is built in
the tree algorithm). No close familiarity with
Min the application domain is required either.
Criterion Seg Size 10% 30% 50% These benefits, and others, have rendered tree
analysis very popular as a data analysis model,
LOGIT 62.4 83.0 89.7
especially in this age of digital marketing, where
FRAC 63.8 77.5 86.9
rules based on decision trees are powering
RFM 34.9 61.2 79.7
many online recommendation engines that de-
CHAID 1 150 42.4 68.3 79.8
termine which product/content to present to a
CHAID 1 300 45.8 68.2 75.9
customer once he/she logs online. Thus, the
CHAID 2 150 59.1 80.7 87.1
availability to generate trees automatically and
CHAID 2 300 61.0 81.3 87.7
inexpensively opens up new frontiers for using
GA-3 4 150 60.0 73.1 81.7
tree classifiers to analyze and understand the
GA-3 4 300 60.7 72.1 79.9
relationship between data elements in a data set
GA-4 4 150 59.1 69.0 77.8
and translate them into rules for making deci-
GA-4 4 300 60.0 77.0 85.0
sions.
STA-1 3 150 54.5 82.4 87.4
STA-1 3 300 50.9 82.4 88.3
STA-1 4 150 61.1 75.8 83.4 APPENDIX A: SPLITTING CRITERIA
STA-1 4 300 60.4 76.4 84.3
Splitting criteria are used to determine the best
STA-2 3 150 57.5 80.0 86.5
split, out of the many possible ways to partition
STA-2 3 300 50.7 81.8 90.3
a node. By and large, we can classify the splitting
LOGIT ⫽ Logistic regression criteria into two families: one based on the
FRAC ⫽ FRAC-based segmentation value of the node, the other on the value of the
RFM ⫽ RFM-based segmentation
partition.
CHAID ⫽ CHAID tree
STA-1 ⫽ STA tree with 1-variable split
STA-2 ⫽ STA tree with 2-variable split Node-Value-Based Criteria
GA-3 ⫽ GA tree with 3-variable split
GA-4 ⫽ GA tree with 4-variable split
These criteria are based on the value of the
node. The objective is to maximize the improve-
ment in the node value that results by splitting
the automatic tree classifiers outperform the a node into two or more splits.
RFM and FRAC methods, and come only short The value of a node t is a function of the
of the logistic regression results. The practical response rate (RR) of the node, RR(t) (i.e., the
implication of these results is that automatic proportion of buyers). The “ideal” split is the
trees may be used as a substitute to judgmen- one that partitions a father node into two chil-
tally based methods, even to logistic regression dren nodes, one that contains only buyers, i.e.,
models. We note, however, that these observa- RR(t) ⫽ 1, and the other only nonbuyers, RR(t)
tions may be specific to the application in- ⫽ 0. Clearly, in applications where the number
volved, and while it may hold for the collectible of “responders” is more or less equal to the
industry, it may not for other industries. number of “nonresponders,” the “worst” split is
Clearly, decision trees have clear benefits the one that results in two children each having
over logistic regression from the point of view of about the same proportion of buyers and non-
the users. Trees are easy to understand and buyers, i.e., RR(t) d 1/2.
interpret, if, of course, properly controlled to We define the value (or quality) of a node t as
avoid unbounded growth. The output of deci- a function Q(RR). In the case where the propor-
sion trees can be presented by means of rules, tion of buyers in the population is about one-
which are clearly related to the problem. Unlike half, Q(RR) satisfies the following conditions:

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

14
 PREDICTIVE MODELING

● Max Q(RR) ⫽ Q(0) ⫽ Q(1) But since Yi is a binary variable (1 buy, 0 no-
● Min Q(RR) ⫽ Q(1/2) buy),

冘Y ⫽冘Y ⫽B
● Q(RR) is a concave function of RR (i.e., the
2
second derivative is positive). i i

● Q(RR) is symmetric, i.e., Q(RR) ⫽ Q(1

i i

⫺ RR). Y៮ ⫽ 冘 Y /共B ⫹ N兲 ⫽ RR
i
i

The first two conditions stem from our defi-

nition of the “best” and the “worst” splits; the
concavity condition follows from the first two
conditions; and the symmetry condition from
the fact that the reference point is 1⁄2.
Clearly, there are many functions that satisfy
these requirements. Below we provide several
examples for the case of a two-way split:
a. The (piecewise) linear function, e.g.,
And we obtain:
Var 共Y兲 ⫽ RR ⫺ RR2 ⫽ .25 ⫺ 共RR ⫺ .5兲2
Which is a quadratic node-value function. Note
that this function satisfies all the conditions for
the function Q(RR) mentioned above.
c. The entropy function (Michalski, Carbonell,

再
& Mitchell, 1993)
1 ⫺ RR RR ⱕ 1/ 2
Q共RR兲 ⫽ RR RR ⬎ 1/ 2 (A.1)
Q共RR兲 ⫽ ⫺关RR log 共RR兲 ⫹ 共1 ⫺ RR兲
b. The quadratic function ⫻ log 共1 ⫺ RR兲兴 (A.3)

Q共RR兲 ⫽ a ⫹ b 䡠 RR ⫹ c 䡠 RR 2 The entropy is a measure of the information

content, the larger the entropy, the better.
where a, b, and c are parameters. Hence the best split is the one with the largest
One can show that the only quadratic func- entropy of all possible splits of a node.
tion that satisfy all these conditions (up to a Figure A.1 exhibits all the three functions
constant) is: graphically.
Now, the node value resulting by partitioning
Q共RR兲 ⫽ RR共RR ⫺ 1兲 (A.2) a node into two children nodes is obtained as
the average of the node value of the two descen-
dant nodes, weighted by the proportion of cus-
We note that the variance criterion used by AID tomers, i.e.:
to define the value of a node, in the binary
yes/no case, is a quadratic node value function.
To show this, we evaluate the variance of the N1 N2
Q共RR 1兲 ⫹ Q共RR 2兲 (A.4)
choice variable Y. Denoting by: N N

Yi — the choice value of observation i where:

Y៮ — the mean value over all observations
B — the number of buyers N – the number of customers in the father
node
N — the number of nonbuyers
N1, N2 – the number of customers in the

Var 共Y 兲 ⫽ 冘 共Y ⫺ Y៮ 兲 /共B ⫹ N兲
i
i
2
descendant left node (denoted by the index
1) and the right node (denoted by the index
2), respectively. In the following we always
⫽ 冘 Y /共B ⫹ N兲 ⫺ Y៮
2
i
2 assume N1 ⬍ N2 (i.e., the left node is the
i smaller one).

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

15
 JOURNAL OF INTERACTIVE MARKETING
FIGURE A.1.
Node value functions.
RR1, RR2 – the response rates of the left and
the right nodes, respectively.
Q(RR1), Q(RR2) – the corresponding node
value functions.
Thus, the improvement in the node value
resulting by the split is given as the difference:
N1 N2
Q共RR 1兲 ⫹ Q共RR 2兲 ⫺ Q共RR兲 (A.5)
N N
And we seek the split that yields the maximal
improvement in the node value. But since, for a
given father node, Q(RR), is the same for all
splits, Eq. (A.5) is equivalent to maximizing the
node value (A.4).
Clearly in DBM applications where the num-
ber of buyers is largely outnumbered by the
number of nonbuyers, the reference point of 1⁄2
may not be appropriate. A more suitable refer-
ence point to define the node value is TRR,
where TRR is the overall response rate of the
JOURNAL OF INTERACTIVE MARKETING
training audience. Another alternative is the
cutoff response rate, CRR, calculated based on
economic criteria. The resulting node value
functions in this case satisfy all conditions
above, except that they are not symmetrical.
Now depending upon the value function
Q(RR), this yields several heuristic criteria for
determining the best split. For example, for the
piecewise linear function and an hierarchical
tree, a possible criterion is choosing the split
that maximizes the response rate of the smaller
child node, Max {RR1, 1 ⫺ RR1}. In a binary tree,
the split which yields the most difference in the
response rates of the two descendant nodes. In
the quadratic case, a reasonable function is (RR
⫺ TRR)2. Or one can use the entropy function
(A.3).
Finally, we note that with a concave node
value function, basically any split results in a
positive value improvement, however small.
Thus, when using the node-value based criteria
for determining the best split, it is necessary to
● VOLUME 15 / NUMBER 2 / SPRING 2001
16
 PREDICTIVE MODELING

T A B L E A.1
Components of Y

Split Buyers Non-Buyers Total

1 Observed B1 N1 T1 ⫽ B1 ⫹ N1
Expected T 1 䡠 B/T T 1 䡠 N/T
2 Observed B2 N2 T2 ⫽ B2 ⫹ N2
Expected T 2 䡠 B/T T 2 䡠 N/T
3 Observed B3 N3 T3 ⫽ B3 ⫹ N3
Expected T 3 䡠 B/T T 3 䡠 N/T
Total B ⫽ B1 ⫹ B2 ⫹ B3 N ⫽ N1 ⫹ N2 ⫹ N3 T ⫽ T1 ⫹ T2 ⫹ T3

impose a threshold level on the minimum seg- One can find the Z value corresponding to the
ment size and/or the minimum improvement P_ value, denoted Z1-␣/2, using:
in the node value, or otherwise the algorithm
will keep partitioning the tree until each node Abs共RR 1 ⫺ RR 2兲

冑
contains exactly one customer. Z 1⫺␣/ 2 ⫽
RR 1 ⴱ 共1 ⫺ RR 1兲/共B 1 ⫹ N 1
Partition-Value-Based Criteria ⫹ RR 2 ⴱ 共1 ⫺ RR 2兲/共B 2 ⫹ N 2兲
(A.6)
Instead of evaluating nodes, one can evaluate
partitions. A partition is considered as a “good”
one if the resulting response rates of the chil- where:
dren nodes are significantly different than one RR1, RR2 – the response rates of the left child
another. This can be expressed in terms of test node (denoted by the index 1) and the right
of hypothesis. For example, in a two-way split child node (denoted by the index 2) respec-
case: tively
B1, B2 – the corresponding number of buyers
H 0: p 1 ⫽ p 2
N1, N2 - the corresponding number of non-
H 1: p 1 ⫽ p 2 buyers
and then extract the P_value from a normal
where p1 and p2 are the true (but unknown) distribution table.
response rates of the left child node and the
right child node, respectively. (b) The chi-square test
A common way to test the hypothesis is by In the case of a multiple split, the test of
calculating the P_ value, defined as the proba- hypothesis is conducted by means of the chi
bility to reject the null hypothesis, for the given square test of independence.
sample statistics, if it is true. Then, if the result- The statistic for conducting this test, denoted
ing P_ value is less than or equal to a predeter- by Y, is given by:
mined level of significance (often 5%), the hy-

冘 共Observed ⫺ Expected兲2
pothesis is rejected; otherwise, the hypothesis is
accepted. Y⫽ (A.7)
splits
Expected
(a) The normal test
The hypothesis testing procedure draws on
the probability laws underlying the process. In Table A.1 exhibits the calculation of the com-
the case of a two-way split, as above, the hypoth- ponents of Y for a three-way split, extending the
esis can be tested using the normal distribution. notation above to the case of three child nodes.

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

17
 JOURNAL OF INTERACTIVE MARKETING
This table can easily be extended to more than
three splits per node. Y is distributed according
to the chi-square distribution with (k ⫺ 1) de-
grees of freedom, where k is the number of
splits for the node. One can then extracts the
P_value for the resulting value of Y from the chi
square distribution. The best split is the one
with the smallest P_value.
(c) The smallest child test
Finally, this criterion is based on testing the
hypothesis:
H 0: p ⬎ TRR
H 1: p ⱕ TRR
where p here stands for the true (but unknown)
response rate of the smaller child node, and
TRR is the observed response rate for the train-
ing audience.
To test this hypothesis we define a statistic Y
denoting the number of standard deviations
(“sigmas”) that the smaller segment is away
from TRR, i.e.:
RR ⫺ TRR
Y⫽
冑RR共1 ⫺ RR兲/N
where RR is the observed response rate of the
smaller child node, and N the number of obser-
vations.
Large values of Y mean that p is significantly
different than TRR, indicating a “good” split.
For example, one may reject the null hypothe-
sis, concluding that the split is a good one, if Y
is larger than 2 “sigmas.”
APPENDIX B: TREE CLASSIFIERS
We discuss below the three tree classifiers that
were involved in our study—STA, CHAID, and
GA.
STA—Standard Tree Algorithm
STA is an AID-like algorithm. The basic AID
algorithm is a univariate binary tree. In each
iteration, each undetermined node is parti-
tioned based on one variable at a time into two
JOURNAL OF INTERACTIVE MARKETING
descendant nodes. The objective is to partition
the audience into two groups that exhibit sub-
stantially less variation than the father node.
AID uses the sum of squared deviations of the
response variable from the mean as the mea-
sure of the node value, which, in the binary
yes/no case, reduces to the minimum variance
criterion, (RR ⫺ 0.5)2, where RR is the response
rate (the ratio of the number of responders to
the total number of customers) for the node
(see also Appendix A).
In each stage, the algorithm searches over all
remaining predictors, net of all predictors that
had already been used in previous stages to split
father nodes, to find the partition that yields the
maximal reduction in the variance.
In this work we have expanded the AID algo-
rithm in two directions:
● Splitting a node based on two predictors at
a time to allow one to also account for the
interaction terms to affect the tree struc-
ture.
● Using different reference points in the
evaluation criterion that are more appro-
priate for splitting populations with
marked differences between responders
and nonresponders. Possible candidates
are the overall response rate of the training
audience, or even the cutoff response rate
separating between targets and nontargets.
We therefore refer to our algorithm as STA
(Standard Tree Algorithm) to distinguish it
from the conventional AID algorithm.
CHAID
CHAID (Chi-Square AID) is the most common
of all tree classifiers. Unlike AID, CHAID is not
a binary tree as it may partition a node into
more than two branches. CHAID categorizes all
independent continuous and multivalued inte-
ger variables by “similarity” measures, and con-
siders the resulting categories for a variable as a
whole unit (group) for splitting purposes. Take
for example the variable MONEY (money
spent) that is categorized into four ranges, each
is represented by a dummy 0/1 variable which
assumes the value of 1 if the variable value falls
● VOLUME 15 / NUMBER 2 / SPRING 2001
18
 PREDICTIVE MODELING
in the corresponding range, 0-otherwise. De-
note the resulting four categorical variables as
variables A, B, C and D, respectively. Since
MONEY is an ordinal variable there are three
possibilities to split this variable into two adja-
cent categories: (A, BCD), (AB, CD), (ABC, D);
three possibilities to split the variable into three
adjacent categories: (A, B, CD) (AB, C, D), (A,
BC, D); and one way to split the variable into
four adjacent categories (A, B, C, D). Now,
CHAID considers each of these partitions as a
possible split, and seeks the best combination to
split the node from among all possible combi-
nations. As a result, a node in CHAID may be
partitioned into more than two splits, as many
as four splits in this particular example. The
best split is based on a chi-square test, which is
what gave this method its name.
Clearly, there are many ways to partition a
variable with K values into M categories (chil-
dren nodes). To avoid choosing a combination
that randomly yields a “good” split, some ver-
sions of CHAID use an adjusted P_ value crite-
rion to compare candidate splits.
Let L denote the number of possible combi-
nations to combine a variable with K values into
M categories.
Let ␣ denote the Type-I error (also known as
the level of significance) in the chi-square test
for independence. ␣ is the probability of reject-
ing the null hypothesis that there is no signifi-
cant difference in the response rates of the
resulting child nodes, when the null hypothesis
is true.
Now, the probability to accept the null hy-
pothesis in one combination is (1 ⫺ ␣), and in
L successive combinations (assuming the test of
hypotheses are independent) is (1 ⫺ ␣)L.
Hence the probability of making a Type-I error
in at least one combination is 1 ⫺ (1 ⫺ ␣)L,
which is greater than ␣.
To yield a “fair” comparison of the various
combinations, ␣ is replaced by the resulting P_
value. In most cases, the P_ value is very small,
and we therefore can use the approximation
1 ⫺ 共1 ⫺ P_value兲L ⬇ L 䡠 P_value
The resulting quantity, L 䡠 P_ value is the ad-
JOURNAL OF INTERACTIVE MARKETING
justed P_value, and L is referred to as the Bon-
ferroni multiplier. Each combination yields a
different adjusted P_ value. The “best” combi-
nation to partition the node by is the one that
yields the smallest adjusted P_ value.
The number of possibilities for combining a
variable with K values into M categories de-
pends on the type of the variable involved. In
our algorithm, we distinguish among three cas-
es:
● Ordinal variables where one may combine
only adjacent categories (as in the case of
the variable MONEY above).
● Ordinal variables with a missing value that
may be combined with any of the other
categories.
● Nominal variables where one may combine
any two (or more) values, including the
missing value (e.g., the variable MARITAL
with four nominal values: M - married, S -
single, W - widow, D - divorce).
Table B.1 exhibits the number of combina-
tions for several representative values of K and
M.
Genetic Algorithm (GA)
All tree algorithms described above are combi-
natorial, in the sense that in each stage they go
over all possible combinations to partition a
node. This number gets excessively large even
for one-variable splits and becomes computa-
tionally prohibitive with multivariable splits.
Consequently, all tree algorithms are univariate
(AID, CHAID) or at best bivariate (STA). Yet, it
is conceivable that splits based on several vari-
ables at a time (more than 2) may be more
“homogeneous” and therefore better from the
standpoint of segmentation. Thus, by confining
oneself to using only univariate and even biva-
riate tree algorithms, one may miss out the
better splits that could have been obtained oth-
erwise with multivariate algorithms.
To resolve this issue, we developed a Genetic
Algorithm (GA) tree for segmentation, which,
unlike the other trees, is a non-combinatorial
algorithm in the sense that it employs a ran-
dom, yet a systematic search approach to grow a
● VOLUME 15 / NUMBER 2 / SPRING 2001
19
 JOURNAL OF INTERACTIVE MARKETING

which the strongest and the fittest have a higher

T A B L E B.1
likelihood of reproduction than the weak and
Number of Possible Combinations
the unfit. Thus, the succeeding descendants,
Ord ⫹ having inherited the better properties of their
K M Ordinal Miss Nominal parents, tend to be even stronger and healthier
than their predecessors and therefore get im-
2 2 1 1 1 proved over time with each additional genera-
3 2 2 3 3 tion (Davis, 1991).
4 2 3 5 7 This idea has been applied to find heuristic
4 3 3 5 6 solutions for large-scale combinatorial optimiza-
5 2 4 7 15 tion problems. Using the terminology of GA,
5 3 6 12 25 each solution to the optimization problem cor-
5 4 4 7 10 responds to a chromosome, each variable to a
6 2 5 9 31 gene, and its value to an allele.
6 3 10 22 90 Starting with the “better” solutions in each
6 4 10 22 65 generation (according to some “fitness” mea-
6 5 5 9 15 sure), GA creates successive offspring solutions
7 2 6 11 63 that are likely to result in a better value for the
7 3 15 35 301 objective function as one goes from one gener-
7 4 20 50 350 ation to the other, thus finally converging to a
7 6 6 11 21 local, if not a global optimum (Holland, 1975).
8 2 7 13 127 These solutions are created by means of a “re-
8 4 35 95 1701 production” process involving two basic opera-
8 6 21 51 266 tions: “mutations” and “crossovers.”
8 7 7 13 28
9 2 8 15 255 ● Mutation – randomly changing some of the
9 4 56 161 7770 variables (genes) of the parent solution.
9 6 56 161 2646 ● Crossover – crossing over variables (genes)
9 8 8 15 36 of two parent solutions, some are taken
10 2 9 17 511 from the “mother” solution, the rest from
10 4 84 252 34105 the “father” solution.
10 6 126 406 22827
10 8 36 92 750 In the context of our segmentation problem,
10 9 9 17 45 GA is used as an algorithm to grow the tree and
generate candidate splits for a node. A solution
in our case is a collection of splitting rules spec-
tree, rather than go over all possible number of ifying whether a customer belongs to the left
combinations. This significantly reduces the segment or to the right segment. For example,
number of combinations to consider in parti- if X3 ⫽ 1, X4 ⫽ 0, and X7 ⫽ 1, the customer
tioning a node, thus allowing one to increase belongs to the left segment, otherwise he/she
the number of variables to split a node beyond belongs to the right segment.
two. In fact, with this approach one can theo- One may use several ways to represent splits
retically use any number of variables to split a in GA. One possibility is by means of a vector,
node, but for computational reasons we have the dimension of which is equal to the number
confined the number of simultaneous variables of potential predictors, one entry for each pre-
in this study to either three or four. dictor. The value of each entry indicate how the
Genetic Algorithm (GA) is a general-purpose corresponding predictor affect the split, e.g.,
search procedure, based on the biological prin-
ciple of “the survival of the fittest,” according to 0: X i does not affect the current split

JOURNAL OF INTERACTIVE MARKETING ● VOLUME 15 / NUMBER 2 / SPRING 2001

20
 PREDICTIVE MODELING
⫺1: Xi ⫽ 0 in the current split
1: Xi ⫽ 1 in the current split
In the above example (assuming there are only
10 potential predictors denoted as X1, . . ., X10),
the corresponding vector is given by:
共0, 0, 1, ⫺ 1, 0, 0, 1, 0, 0, 0兲
Now, the crux of the GA method is to define
those descendant solutions from one genera-
tion to the other. These are created in our
algorithm using mutation and crossover opera-
tions, as follows.
Mutation:
● Choose a predictor from the set of predic-
tors in the solution, by drawing a random
number from the uniform distribution
over the range (1 ⫺ V), where V is the
number of potential predictors. Say the
predictor selected is X3.
● Determine the value (allele) of X3 as fol-
lows:
⫺1: with probability of p0
1: with probability of p1
0: otherwise
For example if the allele selected is ⫺1, the
descendant solution becomes
共0, 0, ⫺1, ⫺1, 0, 0, 1, 0, 0, 0兲
and the new split is defined by
X3 ⫽ 0, X4 ⫽ 0, and X7 ⫽ 1
The value of p0 and p1 are parameters of the
algorithm and are set up in advance by the user.
The mutation operator is applied simulta-
neously on g genes at a time, where the g genes
are also determined at random.
Crossover:
● Pick two solutions at random (a “father”
and a “mother”).
JOURNAL OF INTERACTIVE MARKETING
● Select g consecutive genes at random, say
X3 and X4.
● Create two descendant solutions: a “daugh-
ter” and a “son,” by swapping the selected
genes: The “daughter” solution gets her
mother’s genes, except for X3 and X4 which
are inherited from the father; the “son”
solution gets his father’s genes, except for
X3 and X4 which are inherited from the
mother.
The process starts with a pool of solutions
(population), often created in a random man-
ner. The various reproduction methods are ap-
plied to create the descendant solutions of the
next generation. The resulting solutions are
then evaluated based on the partitioning crite-
ria. The best solutions are retained, and control
is handed over to the next generation, and so
on, until all termination conditions are met.
Finally we note that in our GA tree we define
the smaller child node (i.e., the one with the
fewer number of customers) as a terminal node.
This is based on the plausible assumption that
with multivariate-based splits, the resulting
smaller split appears to be homogeneous
“enough” to make it a terminal node. Hence
the resulting GA tree is hierarchical.
APPENDIX C: THE PREDICTOR SETS
To explore the sensitivity of the automatic tree
classifiers to variations in the number and com-
position of predictors, we have experimented
with several predictor sets.
The first set, the affinity set, consists of the
basic attributes of collectible items:
● Themes – 80 binary predictors indicating
whether the customer has bought an item
with the corresponding theme in the past
● Materials – 10 binary predictors indicating
whether the customer has bought an item
with the corresponding material in the past
● Product codes – 99 binary predictors indi-
cating whether the customer has bought an
item with the corresponding product code
in the past
● VOLUME 15 / NUMBER 2 / SPRING 2001
21
 JOURNAL OF INTERACTIVE MARKETING
Since gender is a very important attribute in
the collectible industry, it was also included in
the affinity set.
The second set includes predictors in the
affinity set plus recency predictors representing
the number of months since last purchase bro-
ken down into mutually exclusive and exhaus-
tive time segments (0 – 6 months, 7–12 months,
13–24 months, etc.).
The third set includes the affinity and re-
cency predictors, plus frequency measures rep-
resenting the total number of products pur-
chased by a customer in the past from all media
channels, broken down by major product cate-
gories
Finally, the last set contains all predictors in
the database, including purchases from direct
mail sources by product categories, money
spent on past purchases by product categories,
time since joining in the list, as well as a variety
of other indicators.
Several procedures were used in the study to
collapse the number of predictors for modeling
purposes to a more manageable size.
REFERENCES
Ben-Akiva, M., & Lerman, S.R. (1987). Discrete Choice
Analysis. Cambridge, MA: The MIT Press.
Breiman, L., Friedman, J., Olshen, R., & Stone, C.
(1984). Classification and Regression Trees. Bel-
mont, CA: Wadsworth.
Bult, J.R., & Wansbeek,T. (1995). Optimal Selection for
Direct Mail. Marketing Science, 14, 378 –394.
Davis, L. (Ed.). (1991). Handbook of Genetic Algo-
rithms. New York: Van Nostrand Reinhold.
Haughton, D., & Oulabi, S. (1997). Direct Marketing
Modeling with CART and CHAID. Journal of Direct
Marketing, 11, 42–52.
Holland, J.H. (1975). Adaptation in Natural and Artifi-
JOURNAL OF INTERACTIVE MARKETING
cial Systems. Ann Arbor: University of Michigan
Press.
Kass, G. (1983). An Exploratory Technique for Investi-
gating Large Quantities of Categorical Data. Applied
Statistics, 29.
Kestnbaum, R.D., (1998). Kestnbaum & Company, Chi-
cago, Private Communication.
Levin, N., & Zahavi, J. (1996). Segmentation Analysis
with Managerial Judgment. Journal of Direct Mar-
keting, 10, 28 – 47.
Long, J.S. (1997). Regression Models for Categorical
and Limited Dependent Variables. Thousand Oaks,
CA: Sage Publications.
Michalski, R.S., Carbonell, J.G., & Mitchell, T.M.
(1983). Machine Learning—An Artificial Intelli-
gence Approach. Palo Alto, CA: Tioga Publishing
Company.
Morwitz, G.V., & Schmittlein, D. (1992). Using Segmen-
tation to Improve Sales Forecasts Based on Purchase
intent: Which “Indenders” Actually Buy? Journal of
Marketing Research, 29, 391– 405.
Morwitz, G.V., & Schmittlein, D. (1998). Testing New
Direct Marketing Offerings: The Interplay of Man-
agement Judgment and Statistical Models. Manage-
ment Science, 44, 610 – 628.
Murthy, K.S. (1998). Automatic Construction of Deci-
sion Trees from Data: A Multi-disciplinary Survey.
Data Mining and Knowledge Discovery, 2, 45–389.
Novak, P.T., de Leeuw, J., & MacEvoy, B. (1992). Rich-
ness Curves for Evaluating Market Segmentation.
Journal of Marketing Research, 29, 254 –267.
Quinlan, J.R. (1986). Induction of Decision Trees. Ma-
chine Learning, 1, 81–106.
Quinlan, J.R. (1993). C4.5: Program for Machine
Learning. CA., Morgan Kaufman Publishing.
Shepard, D. (Ed.). (1995). The New Direct Marketing.
New York: Irwin Professional Publishing.
Sonquist, J., Baker, E., & Morgan, J.N. (1971). Search-
ing for Structure. Ann Arbor: University of Michi-
gan, Survey Research Center.
Weinstein, A. (1994). Market Segmentation. New York:
Irwin Professional Publishing.
● VOLUME 15 / NUMBER 2 / SPRING 2001
22