Framework for integration of domain knowledge into logistic

regression
Sandro Radovanović Boris Delibašić Miloš Jovanović
University of Belgrade - Faculty of University of Belgrade - Faculty of University of Belgrade - Faculty of
Organizational Sciences Organizational Sciences Organizational Sciences
Serbia Serbia Serbia
sandro.radovanovic@fon.bg.ac.rs boris.delibasic@fon.bg.ac.rs milos.jovanovic@fon.bg.ac.rs

Milan Vukićević Milija Suknović

University of Belgrade - Faculty of University of Belgrade - Faculty of
Organizational Sciences Organizational Sciences
Serbia Serbia
milan.vukicevic@fon.bg.ac.rs milija.suknovic@fon.bg.ac.rs

ABSTRACT
Traditionally, machine learning extracts knowledge solely based on
data. However, huge volume of knowledge is available in other
sources which can be included into machine learning models. Still,
domain knowledge is rarely used in machine learning. We propose
a framework that integrates domain knowledge in form of
hierarchies into machine learning models, namely logistic
regression. Integration of the hierarchies is done by using stacking
(stacked generalization). We show that the proposed framework
yields better results compared to standard logistic regression
model. The framework is tested on the binary classification
problem for predicting 30-days hospital readmission. Results
suggest that the proposed framework improves AUC (area under
the curve) compared to logistic regression models unaware of
domain knowledge by 9% on average.
KEYWORDS
Domain knowledge, Logistic regression, Stacking, Hospital
readmission
ACM Reference format:
Sandro Radovanović, Boris Delibašić, Miloš Jovanović, Milan
Vukićević, and Milija Suknović. 2018. SIG Proceedings Paper in
Word Format: Regular Research Paper. In WIMS ’18: 8th
International Conference on Web Intelligence, Mining and
Semantics, June 25-27, 2018, Novi Sad, Serbia, Jennifer B. Sartor,
Theo D’Hondt, and Wolfgang De Meuter (Eds.). ACM, New York,
NY, USA, 8 pages. https://doi.org/10.1145/3227609.3227653

CCS CONCEPTS 1 INTRODUCTION

• Machine learning → Learning paradigms → Supervised
learning → Supervised learning by classification • Machine
learning algorithms → Ensemble methods
_____________________________________________________
Permission to make digital or hard copies of all or part of this work
for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial
advantage ant that copies bear this notice and the full citation on
the first page. Copyrights for components of this work owned bz
others than ACM to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee. Request permissions
from permissions@acm.org.
WIMS ’18, June 25-27, 2018, Novi Sad, Serbia
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-5489-9-18-06…$15.00
https://doi.org/10.1145/3227609.3227653
One of the fastest growing field in computer science is machine
learning. The goal of machine learning is to develop algorithms
which can artificially generate knowledge (induce models) from
experience (data). However, in some domains (such as
bioinformatics, medicine etc.) a vast amount of expert or domain
knowledge is readily available besides the experience stored in
data. This knowledge is also presented in machine readable format,
such as ontologies or hierarchies. It seems reasonable to include
domain knowledge into machine learning models in such manner
that machine learning algorithms understand the context of the data
better or alleviate some assumptions of learning algorithms. This
can assumably lead to better predictive performance, or to more
understandable and consistent models. Typical example is one of
the most commonly used machine learning algorithm called logistic
regression, which assumes a linear relationship between inputs
attributes. Consequently, logistic regression will have high bias
error if a non-linear relationship between input attributes exists.
One can add attribute interactions in order to alleviate this problem,
but the question is which interactions should be added. If domain
knowledge is already available adding attribute interactions which
 WIMS’18, June 2018, Novi Sad, Serbia
utilize the hierarchical interactions between attributes, the logistic
regression model would then be able to solve this problem.
The main hypothesis of this paper is that integrating knowledge
into machine learning models can improve the results of machine
learning models. In this paper we test this hypothesis partially by
integrating Clinical Classifications Software (CCS) hierarchies into
a logistic regression stacking (stacked generalization) framework.
The experiments are performed on real world medical classification
problem where the problem is to predict whether pediatric patients
will be readmitted to hospital 30 days after the discharge. This
problem is one of the most challenging problems in medical
applications of machine learning [30].
The problem on which our approach will be tested is hospital
readmission in 30-days prior to discharge. This is binary
classification problem which is well known in medical applications
of machine learning. It is challenging because of high
dimensionality (problems often have hundreds or thousands of
attributes), sparsity (data at hand are often sparse) and class
imbalance (hospital readmission have high cost, but occurs on
average 7% for pediatric patients [2]).
2 LITERATURE REVIEW
Using domain knowledge as a source of additional information
for machine learning algorithms is not a new idea. The main reason
why this is interesting is the fact that human generated expert
knowledge can sometimes outperform learning algorithms. This is
especially true in areas such as medicine where there are multiple
complex patterns which needs to be interpreted. Sometimes, the
integrated domain knowledge can be indispensable and
irreplaceable, especially when data at hand contain noisy data,
missing data, when working with rare events, or when domain
problem is hard to define and solve. Therefore, efforts are invested
in order to make machine learning algorithms more effective and
efficient [11]. One can alter goal function of learning algorithm in
order to solve problem directly, i.e. instead of minimizing penalty
function one can minimize cost where errors have different cost
associated with it. One way to include domain knowledge is to
include constraints for learning model, i.e. to include
regularizations [16]. Also, domain knowledge is used to obtain new
data using virtual examples [32], [33]. However, domain
knowledge is mostly used in form of heuristics or exact rules for
feature extraction or feature selection.
Introduction of domain knowledge to learning algorithm must
be carefully implemented. Learning algorithms seeks for optimal
solution of search objective (goal) function and domain knowledge
must be in accordance with goal function or if it alters it function
must be optimizable, i.e. have global optima. In other words,
domain knowledge must have three aspects of convergence [38].
First, hypothesis space must have feasible solution. This means that
hypothesis space encompasses an acceptable approximation of goal
function. If hypothesis space have zero acceptable solutions
learning algorithm would not yield a predictive model. Second,
inclusion of domain knowledge must be efficiently implemented.
This requirement suggest that domain knowledge should be used to
reduce search space or help in faster convergence to optimal
2
S. Radovanović et al.
solution. However, search space can be expanded if domain
knowledge would help learning algorithm perform better. Third,
domain knowledge provides tradeoff between computational cost
and accuracy. Namely, domain knowledge should be used in an
interactive learning process where expert should stop learning
when acceptable solution is achieved.
According to [35] domain knowledge and machine learning can
be combined such that 1) domain expert is used to validate
predictive models after machine learning was applied, 2) expert
provides additional constraints to a learning models and 3) experts
and algorithms are performing in turns, namely algorithm present a
model, expert provide feedback to the model. To this date, first type
of domain knowledge and machine learning combination is mostly
used (especially in sensitive applications such as medicine), while
second and third are less used due to time needed for algorithm
development, model creation and model evaluation by domain
expert. Most broadly incorporation of domain knowledge into
machine learning algorithms can be categorized into four groups
[38]. Namely, those are 1) preparation of training examples, 2)
initialization of hypothesis space, 3) altering the search objective
and 4) augmentation of search procedure. All of the groups are
aimed to improve predictive model generality and/or efficiency of
the learning process.
Preparation of training examples aims to expand training
examples and/or enlarges the number of training examples using
human heuristic which is mapped to algorithm for data generation.
Most often creation of these examples are called creation of virtual
examples. The idea is to utilize similarities between concepts which
are labels. This process is very useful in areas where rare events
occurs. Utilization of domain knowledge in example generation in
medical application, and more specifically hospital readmission
problem is presented in [32], [33]. It has been shown that creation
of virtual examples using similarity between diagnoses in ICD-9-
CM hierarchy outperform classic approaches such as
undersampling and oversampling. Additionally, better
performances are obtained if there are very small amount of
examples and if there exists huge class imbalance problem (rare
events problems). The reason why virtual examples work is that
virtual examples are created in areas where density of minority
class is high. In other words, virtual examples are created as an
interpolation of existing examples. Domain knowledge is used as a
guidance in input space. As a major drawback of this process is the
increase in the computational cost of classifier training and loss of
interpretation of model probability score if virtual examples are
used for oversampling.
Initialization of hypothesis space using domain knowledge is
most often used for reducing hypothesis space. One group of papers
belonging to this category is focused on incorporation of domain
knowledge into the kernel methods. The concepts, which present
domain knowledge, are most often presented as a kernel quality
criterion [34] and as such are used for kernel selection. However,
this approach is less used. Another example is argument-based
machine learning (ABML) [9], [22] which present an extension of
machine learning with domain knowledge in form of arguments
with idea of reduction of hypothesis space. The idea is based on the
 Framework for integration of domain knowledge into logistic
regression
fact that domain experts focus on a specific problem when provide
domain knowledge which means that arguments are not general
(exceptions are taken using counter-arguments). Also,
disagreements between experts can be imported into model and
reasoner will select which one of them is acceptable. The benefit of
using arguments is in imposing a constraint over the search space
which will reduce overfitting problem and, additionally, proposed
model will be in accordance with expert knowledge. Arguments
can present positive feedback (example belongs to specific class)
or negative (example does not belong to specific class). As a
drawback of such approach one must use constrained optimization
methods which are not available in standard off-the-shelf tools. In
this category we can add creation of initial hypothesis, which can
be interpreted as a help from domain knowledge in such manner to
provide better starting point and speed up the process of learning.
It is most often used with neural networks where structure of the
network is imposed by domain expert. This research belongs to this
category where domain knowledge in the form of hierarchy is used
as a structure for logistic regression model. More details about
other papers similar to this will be presented in following
paragraph.
The idea of enhancing logistic regression with medical domain
knowledge in form of hierarchies is not new and has already given
better results compared to plain logistic regression. One approach
tried to extract and select attributes using heuristics [27], [26]. It
has been shown in these papers that extracted attributes, not only
improves predictive performance, but also provide better stability
because efficient attribute selection is performed on more general
attributes (represent broader population). Therefore, patterns in
data have higher support and confidence. Additionally, more
general attributes are interpretable to medical expert. Also, it is
worth to notice that these approaches were better in predictive
performance compared to traditional and modern attribute selection
technique. The main reason was the usage of more general
attributes which present general medical concepts. These attributes
were extracted using unsupervised learning methods, namely
logical or operator. Other approach was to utilize domain hierarchy
in form of regularization [14], [13]. Besides better generalizability
of predictive model, these models lead to improved interpretability
of logistic regression model. Additionally, these models can be
used for gaining further insights of causes of readmission and as
risk indicators. Additionally, similar approaches have already
shown that utilization of domain knowledge in form of hierarchy
improve predictive performance of learning algorithm and model
interpretability [40], [4], [6].
Altering search objective is third group of applications where
domain knowledge is of help to machine learning models. This type
of applications modifies predictive model’s optimization problem
either by introducing weights of examples or by imposing costs of
the errors. In latter, wrapper around learning algorithm is applied
which optimizes cost provided by domain expert. Typical example
is MetaCost algorithm [7]. It perform bagging-like procedure
utilizing cost and probability obtained from learning model.
Namely, output of the model will be weighted average of predicted
WIMS’18, June 2018, Novi Sad, Serbia
classes which are computed as minimal values of product of cost
and probability score for class. Other approach, where weight is
given to an example, is also often used. It is considered as a very
hard problem since domain expert has to give weight to each
example. In the era of huge data sets this is time consuming task.
However, one can give weight to a group of examples per specific
value of other feature or per class [39]. Example weighting can be
performed based on decision rules presented in [15]. Having
decision rules in mind domain expert can propose rules and give
weights to examples that satisfy proposed rules.
Most often domain knowledge is used to augment search space.
Augmentation of search space is expansion of existing dataset by
inclusion of logical interactions. One example is First Order
Inductive Learner (FOIL) [25] which is a system that learns Horn
clauses (instead of the classical attribute-value languages) from
relations. Relations are constructed by adding one literal at a time.
In order to evaluate relations information gain is calculated. A
broader field of algorithms which heavily depends on a domain
knowledge is known as an Inductive Logic Programming [19].
These algorithms generates logical theories using examples in data
and domain knowledge. We refer to interested reader to other
examples of ILP such as PROGOL [23], ALEPH [29] and TILDE
[3]. Similar approach called propositionalization [17] which
decouples feature construction phase from model construction
phase is proposed. It is based on relational rule learning for
classification and prediction where relational rules augment feature
space in process called constructive induction. Relational rules
consumes domain knowledge and structural properties of
examples. Arguably, as a downside one can say that ILP does not
warrant optimal solution since most of the approaches uses
heuristics or meta-heuristics such as hill-climbing. However, this
optimization technique is justified since number of combination
would lead to combinatorial explosion. We would like to mention
description logic learners which are used to construct descriptions
of sets of examples and then reason about these descriptions [1].
Although medical and biomedical domain have vast amount of
knowledge which can be exploited in order to provide improve
predictive models there are other applications and utilization of
domain knowledge in machine learning models which can be
classified to initialization of hypothesis space and altering search
objective at the same time. Sometimes a classifier needs to classify
multiple outputs at the same time. This problem is called multi-
label classification. Main idea of multi-label methods is to use
correlation between labels. However, often we can construct
hierarchy or hierarchy can be imposed from domain. When
hierarchy is imposed it can be utilized to improve predictive
performance by hierarchical classification [28]. In other words, we
can predict higher nodes of hierarchy and then specialize to lower
levels and finally leafs of the hierarchy. There are several
3
 WIMS’18, June 2018, Novi Sad, Serbia S. Radovanović et al.

Bacterial infection

Tuberculosis ... Septicemia

Other specified Unspecified

ICD 01000 ... ...
septicemia septicemia

... ... ... ICD 0031 ICD 77181

ICD 1374 ... ... ... ...

ICD 0389 ICD 99592

Figure 1: Excerpt of CCS hierarchy.

algorithms that proved that hierarchical classification improves

predictive performance of a learning algorithms such as Predictive
Clustering Trees (PCT) [20] and HOMER [31]. Table 1: Dataset description
Compared to the existing approaches in literature, we induce the
Diagnoses # rows # %
hierarchical structure into the logistic regression model by using columns readmission
stacking, which is an approach that hasn’t been so far reported in Anemia 3,445 301 29.96%
literature. Esophageal reflux 4,286 301 19.58%
(ER)
3 METHODOLOGY Epilepsy 6,340 232 18.26%
Acute Respiratory 3,610 173 13.32%
The methodology section consists of three parts. First, we Infections (ARI)
explain the hospital readmission datasets used in this research. Asthma 13,907 152 9.54%
Next, we introduce domain hierarchy and logistic regression. Pneumonia 6,931 142 7.62%
Finally, we explain the methodology and experiments in the
experimental setup subsection. A dataset row present one hospital admission, while a column
presents an ICD-9-CM code. Since there are over 15,000 ICD-9-
3.1 Data CM codes (diagnosis), only those ICD-9-CM codes were selected
Data used in this research comes from hospital discharge where over 0.5% of admissions had that specific diagnosis.
records from California, State Inpatient Database (SID), Healthcare Therefore, we reduced the input space by large just by removing
Cost and Utilization Project (HCUP), Agency for Healthcare sparse columns. Values in matrix present whether diagnosis is
Research and Quality [24]. Purpose of this database is to track all present or not (binary values). Finally, it can be observed that
hospital admissions at the individual level, where one patient can hospital readmission is a fairly imbalanced problem having from
have a maximum of 25 diagnoses for one admission. Every 7.62% to 29.96% hospital readmissions.
diagnosis is presented as an ICD-9-CM code. For the purpose of
this paper, several datasets were extracted which contain pediatric 3.2 Domain hierarchy
patients with diagnoses that are considered as the most important
for pediatric subpopulation [2]. Datasets on which the experiments In order to impute domain knowledge into logistic regression
have been done are presented in Table 1. we employed domain hierarchy called Clinical classification
software (CCS) for ICD-9-CM [10] which presents clusters of
clinically meaningful categories. It is created so one can analyze

4
 Framework for integration of domain knowledge into logistic
regression
costs, utilization, and outcomes associated with particular
diagnoses and medical procedures. CCS can be presented in multi-
level form such that diagnoses are presented in four levels, i.e.
hierarchies. However, a hierarchy is not strict, meaning that not
every path on a hierarchy is of depth four. Top level of the hierarchy
presents diagnoses on a general level (i.e. Diseases of the
circulatory system) which can be used mostly for descriptive
analysis. As one goes down the hierarchy diagnoses becomes more
specific and more usable for analysis. On bottom level specific
ICD-9-CM codes are presented. Hierarchical relationship between
levels is presented in “is-a” relationship. An example of a CCS
diagnosis hierarchy is presented below:
7. Diseases of the circulatory system
7.1. Hypertension
7.1.1. Essential hypertension [ICD-9-CM 401.9]
7.1.2. Hypertension with complications and secondary
hypertension [ICD-9-CM 405.9]
7.1.2.1. Hypertensive heart and/or renal disease
7.1.2.2. Other hypertensive complications
7.2. Diseases of the heart
It should be noted that CCS hierarchy is a forest, which means
that multiple hierarchies exist. More specifically, the CCS contains
17 hierarchies which present major diagnoses states.
The CCS hierarchy contains mapping between every single
ICD-9-CM code to leaf nodes in hierarchy. There are over 14,000
ICD-9-CM codes which corresponds to diagnoses and over 3,900
ICD-9-CM codes which corresponds to procedures. Since data we
used have only diagnoses we utilized hierarchy with 14,000 ICD-
9-CM codes. The depth of the hierarchy is variable ranging from
two to four, while depth 3 is most dominant.
3.3 Logistic Regression
Logistic regression is one of the most popular and most widely
used algorithm in machine learning. We can present it [12] as:
𝑝 and child element c. We define set of higher medical concepts S as
log ( ) = θ0 + θ1 𝑥1 + ⋯ + θ𝑛 𝑥𝑛 (1)
1−𝑝
𝑝
where log ( ) present logarithm of odds ratio, 𝜃 weight of
1−𝑝
associated attribute 𝑥. By optimizing 𝜃 such that penalty function
is minimized we can obtain probability that a hospital readmission
will occur. Penalty function is:
𝑚
𝐿(θ) = ∑(1 + exp(−𝑦𝑖 (𝑥 𝑇 θ + 𝑐)) (2)
𝑖=1
where 𝑥 represent input attributes, 𝑦 binary label which takes
values from {−1, 1}, θ weight vector associated to 𝑥, 𝑚 number of
examples and 𝑐 random noise. This loss function present a
maximum likelihood from Formula 1. It is convex and continuous
where function tends to zero as 𝑦𝑖 (𝑥 𝑇 θ + 𝑐) tends to infinity. In
other words if predictions are in good direction then function will
be close to zero. We note that even if model is certain in prediction
WIMS’18, June 2018, Novi Sad, Serbia
there will be small loss. Random noise 𝑐 is normal distributed with
zero mean. Optimization is often performed using gradient descent
methods. After optimization we can calculate probability of
hospital readmission using:
exp(θ0 + θ1 𝑥1 + ⋯ + θ𝑛 𝑥𝑛 )
𝑝(𝑦̅) = (3)
1 + exp(θ0 + θ1 𝑥1 + ⋯ + θ𝑛 𝑥𝑛 )
Formula 3 is obtained as a derivation by 𝑝 from Formula 1. The
probability of hospital readmission 𝑝(𝑦̅) is a value between zero
and one. This formula is present probabilistic interpretation of
Formula 2 and we can say that if i.e. θ0 + θ1 𝑥1 + ⋯ + θ𝑛 𝑥𝑛 = 4
then predicted probability 𝑝(𝑦̅) would be ≈ 0.98 . Similarly, if
θ0 + θ1 𝑥1 + ⋯ + θ𝑛 𝑥𝑛 = −4 then 𝑝(𝑦̅) ≈ 0.02.
However, in order to make prediction we introduce decision
threshold τ. If probability is greater or equal to τ then hospital
admission will be classified as readmission in 30-days, otherwise
not. In this paper value of τ is set to 0.5, which is default value for
decision threshold. It can be argued that this is not an optimal value
of decision threshold however some performance metrics such as
AUC are threshold independent and will be discussed accordingly.
Also, we do not seek to find best performing algorithm but
improvement using stacked generalization with domain knowledge
hierarchy and selection of τ to 0.5 allows the same experimental
setup. Additionally, same threshold allows same evaluation of the
proposed approach and baseline method.
3.4 The stacking logistic regression framework
In this research we propose stacking logistic regression
framework which integrates domain knowledge from CCS which
groups clinically meaningful diagnoses in form of hierarchy.
Integration of domain knowledge into machine learning (in this
case logistic regression) algorithms will be performed using
stacking approach [36] where structure of stacking network will be
CCS hierarchy.
If we have hierarchy H which presents parent-child elements.
Namely, element h from hierarchy H contains of parent element p
non-leaf elements in hierarchy H. Additionally, we can extract
higher medical concept level in hierarchy and sort set in descending
order by level. Then, for each medical concept s we select its
children C and train logistic regression model LR from dataset D
using only columns C. After model training we apply model to the
same dataset D and create new column with name s. We save
logistic regression model LR to a list of models M. The procedure
is repeated for every medical concept s in set of medical concepts
S.
Using Figure 1 we can describe intuition of our approach.
Namely, we create logistic regression model for Tuberculosis using
attributes ICD 01000, …, ICD 1374. Similarly, ICD 0031, …, ICD
0389 are used for learning Other specified septicemia while ICD
77181, …, ICD 99592 are used for learning Unspecified
septicemia. Learning is performed with regard to hospital
readmission within 30-days. Therefore, learning phase produce
probability scores of hospital readmission within 30-days. Obtain
5
 WIMS’18, June 2018, Novi Sad, Serbia
values present values in dataset for attribute which correspond to
that higher order (more genera) medical concept. Further, Other
specified septicemia and Unspecified septicemia are used for
learning more general concept Septicemia. This process is repeated
until the most general concept is learned, in this case Bacterial
infection. Since CCS hierarchy have more hierarchies (recall that
CCS is forest of medical concepts) this procedure is repeated for
every hierarchy. After every hierarchy is learned most general
concepts are used for hospital readmission prediction.
The idea is to group attributes into a hierarchy and gradually
predicting the outcome attribute, instead of predicting the outcome
variable by having all attributes in the same level of hierarchy. This
idea makes sense [18], [5] as many authors have noticed that the
assumption of linearity among attributes does not hold for all
relationship between attributes.
This way of integration of domain knowledge into logistic
regression can be seen as deep feature extraction model, which
resembles neural networks, where domain knowledge provide the
structure of the networks. The benefits of using domain knowledge
are 1) alleviating linear decision boundary problem of logistic
regression, 2) interpretability of the model and 3) more stable
solutions [40]. Also, one can state that this is mixture of experts
[37] model because each feature is constructed as an independent
expert used for hospital readmission prediction.
Logistic regression is known for linear decision boundary.
Because of that sometimes it has problem when there are multiple
groups of examples on different sides of attribute space. In the
literature one can find several proposed solutions for this problem.
First, most simple one, is to use higher order polynomials and
attribute interactions. Defining higher order polynomial is prone to
overfitting and therefore attribute interaction is more useful.
Defining interactions can be automatic from data or by utilization
of hierarchy by domain expert. This approach already gave
performance improvement in hospital readmission prediction [26].
Second approach involve utilization of kernel methods [41]. Using
kernels yield in nonlinear decision boundary in original input space
by constructing a linear decision boundary in a transformed version
of the original input space. Transformed space can, theoretically,
be infinite. However, this approach is seldom used. Finally, output
of one logistic regression can be used as an input of another logistic
regression (similar to neural network). We utilized latter approach
since we believe that this approach is highly compatible with
domain knowledge which is available and heavily underused.
Since hierarchy is domain created concepts which are created
are from domain and can be interpreted by domain expert. Logistic
regression is considered as a predictive algorithm which is highly
interpretable. Namely, coefficients of logistic regression can be
interpreted in terms of increase or decrease of odds of event (in this
paper hospital readmission). In first step of proposed approach we
obtain coefficients of specific diagnoses. Result of that step is
probability of occurrence of hospital readmission for higher order
concept. The process is repeated and we obtain odds of hospital
readmission of higher concept which is again interpretable as odds
of hospital readmission.
6
S. Radovanović et al.
Stable of produced model is considered as a desirable property
of machine learning models in sensitive application, such as
medical. It has been shown that regularization techniques such as
lasso tends to be unstable if dataset is slightly perturbed in presence
of highly correlated attributes [21]. However, utilizing hierarchy
has shown, both theoretically and practically, more stable models
[40].
3.5 Experimental Setup
As a baseline classifier logistic regression will be used. The
diagnoses (bottom level of hierarchy) are used as inputs.
Experiments are performed using 10-fold cross validation. This
means that each dataset is split into 10 subsets where 9 are used for
model training and one for testing the predictive performance. The
process is repeated 10 times such that each of the subsets are used
exactly once for testing. Values of predictive performance are
presented as the average value with standard deviation of
performance on the ten test sets. However, one must be careful
interpreting standard deviation since samples are not independent.
Besides classification accuracy, as a predictive performance
measure, we also report the area under the curve (AUC), precision
and recall. AUC can be interpreted as a probability that a random
positive example (hospital readmission within 30-days) have
higher probability score than random negative example (patient did
not readmit to hospital within 30-days). Precision, which present
percentage of predicted hospital readmission which are truly
hospital readmission, and recall, which is percentage of true
hospital readmission that has been predicted as a hospital
readmission. For all performance measure mentioned above higher
value is better. The decision threshold for the logistic regression
model was set to 0.5 in order for each predictive model to be
comparable.
We note that datasets used in experimental setup differs in
number of examples and columns. Additionally, in each dataset
there is class imbalance which range from 7.62% of readmitted
patients to 29.96% of readmitted patients. Description of the dataset
is presented in Table 1. Hierarchy used for stacked learning is CCS
for ICD-9-CM, as explained in Section 3.2.
4 RESULTS
The experimental results are presented in Table 2. As stated in
the experimental setup section, AUC, Classification accuracy (CA),
Precision, and Recall are reported. Logistic regression, used as a
baseline method is denoted with Basic, while the newly proposed
method is denoted Stacked Domain Hierarchy (SDH). We note that
we could select decision threshold such that CA, Precision and
Recall would be optimized. However, this would lead to
performances which would not be comparable.
 Framework for integration of domain knowledge into logistic
WIMS’18, June 2018, Novi Sad, Serbia
regression

Table 2: Experimental evaluation of proposed model for Having performances improvement in mind we can conclude
hospital readmission prediction that introducing more general concepts and making predictions
based on them does improve predictive performance of logistic
Dataset Method AUC CA Prec. Rec regression. As we can observe by combining Table 1 and Table 2
Anemia Basic 0.696 ± 0.720 ± 0.547 ± 0.375 ± size of the dataset is not correlated with predictive performance.
0.030 0.027 0.069 0.044
Therefore, the main reason for performance improvement is
Anemia SDH 0.722 ± 0.747 ± 0.638 ± 0.365 ±
0.037 0.023 0.061 0.038 introduction of non-linear decision boundary. This leads us that
ER Basic 0.599 ± 0.790 ± 0.390 ± 0.131 ± interaction of diagnoses on higher concepts have influence which
0.035 0.012 0.060 0.021 cannot be seen just by inspecting specific diagnoses. Additionally,
ER SDH 0.618 ± 0.797 ± 0.435 ± 0.112 ± since standard deviation are smaller in SDH models we can say that
0.034 0.014 0.103 0.029 stacking logistic regression framework improve model stability as
Epilepsy Basic 0.601 ± 0.813 ± 0.426 ± 0.070 ± it is shown in [40].
0.019 0.005 0.124 0.021
Epilepsy SDH 0.602 ± 0.818 ± 0.506 ± 0.063 ±
5 CONCLUSION
0.018 0.014 0.101 0.019
ARI Basic 0.569 ± 0.357 ± 0.238 ± 0.766 ± In this paper we proposed the integration of domain knowledge
0.111 0.355 0.179 0.377 integration in logistic regression models on real world medical data.
ARI SDH 0.749 ± 0.861 ± 0.406 ± 0.109 ± The reported results show that the proposed framework consistently
0.019 0.020 0.100 0.049 improves the predictive performance of the logistic regression
Asthma Basic 0.500 ± 0.095 ± 0.095 ± 1.000 ± models. The proposed framework utilizes domain knowledge in
0.000 0.005 0.005 0.000 form of hierarchies, i.e. forests, to perform stacking. Since
Asthma SDH 0.717 ± 0.901 ± 0.311 ± 0.034 ±
hierarchy is used we can say that our framework is a type of mixture
0.029 0.005 0.144 0.019
of experts model. Contributions of this papers are methodological,
Pneumonia Basic 0.649 ± 0.920 ± 0.271 ± 0.029 ±
0.052 0.006 0.159 0.020 namely framework that allows integration domain or expert
Pneumonia SDH 0.685 ± 0.922 ± 0.399 ± 0.033 ± knowledge into logistic regression. Integration is performed in such
0.046 0.006 0.310 0.025 a manner that hierarchies extend the classical logistic regression in
a stacking framework. One can say that this methodology can be
It can be noticed that the SDH model is outperforming the basic seen as a neural network where domain knowledge provide the
model on AUC and classification accuracy on every dataset. More structure of the network. It is supposed to perform better compared
specifically, on average the SDH models have 9.58% better AUC to traditional approach because of the ensemble nature of
compared to the basic model. Other performance measures, except framework. Additionally, proposed framework allows non-linear
Recall, are also better using proposed SDH model having on problems to be solved and more stable solutions (less prone to
average 14.37% better classification accuracy, 12.13% precision, overfitting).
while recall is 31.93% worse. It is worth mentioning that standard However, further experimentation, both on simulated and on
deviation of AUC performances and other performance measures real world datasets, are needed in order to validate the quality of
(although it cannot be directly interpreted as standard deviation) are integration of domain knowledge in form of hierarchies (forests)
smaller. into proposed stacking framework. Possible option of validation is
Further inspection of Table 2 lead to the dataset for which a combination of randomly chosen attributes into a proposed
improvement is not significant. This dataset is Epilepsy. With framework in order to test whether the results are being improved
inspection of the papers regarding hospital readmission among because of the domain knowledge, or because of the stacking
epilepsy patient in pediatric hospitals it has been shown that patient framework. Also, we can test hierarchical clustering in order to
characteristics such as prior health care utilization, demographics, obtain hierarchy (or forest). Namely, we can extract hierarchies
insurance status, social history, patient engagement (i.e. missed from data using i.e. AGNES or DIANA algorithms and use those
appointments) and use of specialty care are most important features hierarchies instead of domain knowledge based hierarchies. If SDH
for predictions [8]. All of those features are not available in framework outranks the other approaches we could be absolutely
diagnoses codes in ICD-9-CM hierarchy. However, we can see sure that expert knowledge makes the difference, and not the
dramatic improvement for Asthma dataset. Inspection of methodology itself. Further experimentation are left for future
classification accuracy and other metric suggest that decision studies.
threshold was set too high to classify example as hospital
readmission. However, basic logistic regression model achieved REFERENCES
AUC 0.500 which means that predictive model was good as [1] Franz Baader, Diego Clavanese, Deborah McGuinness, Daniele Nardi, and
Peter F. Patel-Schneider (Eds.). 2003. The description logic handbook: Theory,
random model. Significant improvement in terms of AUC are also implementation and applications. Cambridge university press.
presented in Acute Respiratory Infections dataset. [2] Jay G. Berry, Sara L. Toomey, Alan M. Zaslavsky, Ashish K. Jha, Mari M.
Nakamura, David J. Klein, Jeremy Y. Feng, Shanna Shulman, Vincent W.
Chiang, William Kaplan, Matt Hall, and Mark A. Schuster. 2013. Pediatric

7
 WIMS’18, June 2018, Novi Sad, Serbia
readmission prevalence and variability across hospitals. JAMA 309, 4 (2013),
372-380.
[3] Hendrick Blockeel, Sašo Džeroski, and Jasna Grbović. 1999. Simultaneous
prediction of multiple chemical parameters of river water quality with TILDE.
In European Conference on Principles of Data Mining and Knowledge
Discovery, Springer, Berlin, Heidelberg, 32-40.
[4] Marko Bohanec, and Boris Delibašić. 2015. Data-mining and expert models for
predicting injury risk in ski resorts. In International conference on decision
support system technology, Springer, Cham, 46-60.
[5] Marko Bohanec, and Vladislav Rajkovič. 1990. DEX: An expert system shell
for decision support. Sistemica 1, 1 (1990), 145-157.
[6] Boris Delibašić, Sandro Radovanović, Miloš Jovanović, Marko Bohanec, and
Milija Suknović. 2018. Integrating knowledge from DEX hierarchies into a
logistic regression stacking model for predicting ski injuries. Journal of
Decision Systems, (2018), 1-8.
[7] Pedro Domingos. 1999. Metacost: A general method for making classifiers cost-
sensitive. In Proceedings of the fifth ACM SIGKDD international conference on
Knowledge discovery and data mining (1999), ACM, 155-164.
[8] Zachary M. Grinspan, Anup D. Patel, Baria Hafeez, Erika L. Abramson, and
Lisa M. Kern. 2018. Predicting frequent emergency department use among
children with epilepsy: A retrospective cohort study using electronic health data
from 2 centers. Epilepsia 59, 1 (2018), 155-169.
[9] Vida Groznik, Matej Guid, Aleksander Sadikov, Martin Možina, Dejan
Georgiev, Veronika Kragelj, Simo Ribarič, Zvezdan Pirtošek, and Ivan Bratko.
2013. Elicitation of neurological knowledge with argument-based machine
learning. Artificial intelligence in medicine 57, 2 (2013), 133-144.
[10] Healthcare Cost and Utilization Project. 2011. Clinical classifications software
(CCS) for ICD-9-CM. Available at: www.hcup-
us.ahrq.gov/toolssoftware/ccs/ccs. jsp. Accessed January, 23, 2018.
[11] Andreas Holzinger. 2016. Interactive machine learning for health informatics:
when do we need the human-in-the-loop?. Brain Informatics 3, 2 (2016), 119-
131.
[12] Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. An
introduction to statistical learning. Vol. 112. New York: Springer.
[13] Miloš Jovanovic, Sandro Radovanovic, Milan Vukicevic, Sven Van Poucke,
and Boris Delibasic. 2016. Building interpretable predictive models for pediatric
hospital readmission using Tree-Lasso logistic regression. Artificial intelligence
in medicine 72 (2016), 12-21.
[14] Iman Kamkar, Sunil K. Gupta, Dinh Phung, and Svetha Venkatesh. 2015. Stable
feature selection for clinical prediction: Exploiting ICD tree structure using
Tree-Lasso. Journal of Biomedical Informatics 53 (2015), 277-290.
[15] Branko Kavsek, and Nada Lavrac. 2004. Analysis of example weighting in
subgroup discovery by comparison of three algorithms on a real-life data set. In
Proceedings of the 15th European conference on machine learning and 8th
European conference on principles and practice of knowledge discovery in
databases (2004). 64-76.
[16] Seyoung Kim, and Eric P. Xing. 2010. Tree-guided group lasso for multi-task
regression with structured sparsity. In Proceedings of the 27th International
Conference on Machine Learning, (2010).
[17] Stefan Kramer, Nada Lavrač, and Peter Flach. 2001. Propositionalization
approaches to relational data mining. In Relational data mining, Springer,
Berlin, Heidelberg, 262-291.
[18] Niels Landwehr, Mark Hall, and Eibe Frank. 2005. Logistic model trees.
Machine learning 59, 1-2 (2005), 161-205.
[19] Nada Lavrac, and Sašo Dzeroski. 1994. Inductive Logic Programming. In WLP
(1994), 146-160.
[20] Gjorgji Madjarov, Dragi Kocev, Dejan Gjorgjevikj, and Sašo Džeroski. 2012.
An extensive experimental comparison of methods for multi-label learning.
Pattern recognition 45, 9 (2012), 3084-3104.
[21] Nicolai Meinshausen, and Peter Bühlmann. 2010. Stability selection. Journal of
the Royal Statistical Society: Series B (Statistical Methodology) 72, 4 (2010),
417-473.
[22] Sanjay Modgil, Francesca Toni, Floris Bex, Ivan Bratko, Carlos I. Chesnevar,
Wolfgang Dvořák, Marcelo A. Falappa et al. 2013. The added value of
argumentation. In Agreement Technologies, Springer, Dordrecht, 357-403.
[23] Stephen Muggleton. 1995. Inverse entailment and Progol. New generation
computing 13, 3-4 (1995), 245-286.
[24] NIS, HCUP Nationwide Inpatient Sample. 2011. Healthcare cost and utilization
project (HCUP).
[25] John Ross Quinlan. 1990. Learning logical definitions from relations. Machine
learning 5, 3 (1990), 239-266.
[26] Sandro Radovanovic, Milan Vukicevic, Ana Kovacevic, Gregor Stiglic, and
Zoran Obradovic. 2015. Domain knowledge based hierarchical feature selection
for 30-day hospital readmission prediction. In Conference on Artificial
Intelligence in Medicine in Europe, Springer, Cham. 96-100.
[27] Petar Ristoski, and Heiko Paulheim. 2014. Feature selection in hierarchical
feature spaces. In International Conference on Discovery Science, Springer,
Cham. 288-300.
S. Radovanović et al.
[28] Luiz M. Romao, and Julio C. Nievola. 2012. Hierarchical classification of gene
ontology with learning classifier systems. In Ibero-American Conference on
Artificial Intelligence, Springer, Berlin, Heidelberg (2012). 120-129.
[29] Ashwin Srinivasan. 2001. The aleph manual. Available at
http://www.cs.ox.ac.uk/activities/machinelearning/Aleph/aleph.html.
[30] Gregor Stiglic, Petra P. Brzan, Nino Fijacko, Fei Wang, Boris Delibasic,
Alexandros Kalousis, and Zoran Obradovic. 2015. Comprehensible predictive
modeling using regularized logistic regression and comorbidity based features.
PloS one 10, 12, e0144439.
[31] Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2008. Effective
and efficient multilabel classification in domains with large number of labels.
In Proc. ECML/PKDD 2008 Workshop on Mining Multidimensional Data
(MMD’08) 21 (2008). 53-59.
[32] Milan Vukicevic, Sandro Radovanovic, Ana Kovacevic, Gregor Stiglic, and
Zoran Obradovic. 2015. Improving hospital readmission prediction using
domain knowledge based virtual examples. In International Conference on
Knowledge Management in Organizations, Springer, Cham (2015). 695-706.
[33] Mila Vukicevic, Sandro Radovanovic, Gregor Stiglic, Boris Delibasic, Sven
Van Poucke, and Zoran Obradovic. 2016. A Data and Knowledge Driven
Randomization Technique for Privacy-Preserving Data Enrichment in Hospital
Readmission Prediction. In 5th Workshop on Data Mining for Medicine and
Healthcare (2016). 10.
[34] Lei Wang, Yan Gao, Kap L. Chan, Ping Xue, and Wei-Yun Yau. 2005. Retrieval
with knowledge-driven kernel design: an approach to improving SVM-based
CBIR with relevance feedback. In Tenth IEEE International Conference on
Computer Vision, 2005. ICCV 2005 2, IEEE (2005). 1355-1362.
[35] Geoffrey I. Webb, Jason Wells, and Zijian Zheng. 1999. An experimental
evaluation of integrating machine learning with knowledge acquisition.
Machine Learning 35, 1, 5-23.
[36] Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2016. Data
Mining: Practical machine learning tools and techniques. Morgan Kaufmann.
[37] Lei Xu, Michael I. Jordan, and Geoffrey E. Hinton. 1995. An alternative model
for mixtures of experts. In Advances In Neural Information Processing Systems
(1995). 633-640.
[38] Ting Yu, Tony Jan, Simeon Simoff, and John Debenham. 2007. Incorporating
prior domain knowledge into inductive machine learning. Unpublished doctoral
dissertation Computer Sciences.
[39] Bianca Zadrozny, John Langford, and Naoki Abe. 2003. Cost-sensitive learning
by cost-proportionate example weighting. In Third IEEE International
Conference on Data Mining, 2003. ICDM (2003), IEEE. 435-442.
[40] Jiayu Zhou, Zhaosong Lu, Jimeng Sun, Lei Yuan, Fei Wang, and Jieping Ye.
2013. Feafiner: biomarker identification from medical data through feature
generalization and selection. In Proceedings of the 19th ACM SIGKDD
international conference on Knowledge discovery and data mining, ACM
(2013). 1034-1042.
[41] Ji Zhu, and Trevor Hastie. 2005. Kernel logistic regression and the import vector
machine. Journal of Computational and Graphical Statistics 14, 1, 185-205.