Electron Markets (2017) 27:387–398
DOI 10.1007/s12525-017-0245-6
RESEARCH PAPER
The data quality improvement plan: deciding
on choice and sequence of data quality improvements
Dominikus Kleindienst 1
Received: 18 December 2015 / Accepted: 3 January 2017 / Published online: 14 January 2017
# Institute of Applied Informatics at University of Leipzig 2017
Abstract With the rapid growth in the amount of data gener-
ated worldwide, ensuring adequate data quality (DQ) is increas-
ingly becoming a challenge for companies: data are, among
others, required to be timely, complete, consistent, valid, and
accessible. Given this multidimensionality, DQ improvements
(DQIs) need to be purposefully chosen and –as there can be
path dependencies– arranged in an optimal sequence. Thus, this
research contributes to performing the complex multidimen-
sional task of ensuring adequate DQ in an economically rea-
sonable manner by providing a formal decision model for iden-
tifying an optimal data quality improvement plan (DQIP). This
DQIP comprises both an economically reasonable selection
and execution sequence of DQIs based on existing interrela-
tionships between different DQ dimensions. Furthermore, a
comprehensive Monte Carlo simulation provides insights in
implications to put the decision model into operation. For prac-
titioners, the decision model enables efficient allocation of re-
sources to DQIs. The model also gives advice on how to se-
quence DQIs and attracts attention to the complex problem
context of DQ in order to support valid managerial decisions.
Keywords Data quality . Dimensions . Interrelationships .
Improvements
Jel Classification M29
Responsible Editor: Hans-Dieter Zimmermann
* Dominikus Kleindienst
dominikus.kleindienst@fim-rc.de
1
FIM Research Center, Universitätsstraße 12,
86159, Augsburg, Germany
Introduction
Today’s generation has never known a world without cell
phones and the Internet and is now increasingly participating
in business-to-customer interactions (Radant et al. 2014). This
is one of many reasons for the inevitable and rapid growth in
the amount of worldwide generated data (Provost and Fawcett
2013), which, as a result, makes data quality (DQ) a growing
challenge for companies. Accordingly, information and com-
munication technologies yield new data-driven business
models that further intensify data-rich and DQ-reliant interac-
tions between companies and customers. For companies, en-
suring adequate DQ is a manifold task that requires the data to
be up-to-date, complete, consistent, valid, and accessible in
order to improve customer interactions. Given this global
and recurrent challenge and against the background of scarce
resources, data errors need to be addressed by an economics-
driven method. That is, DQ improvements (DQIs) have to be
chosen according to their effectiveness and then applied in an
efficient sequence. Particularly, in large-scale datasets, on
which Big Data technologies conduct more and more data
intensive processing (Vera-Baquero et al. 2015), an optimal
choice and sequence of DQIs is – looking at all sorts of ad-
dressable DQ dimensions – not apparent initially. Although
most data errors in a business context are not grave and
Redman (2004) states, Bthat most data errors just cause an
invoice to be incorrect^ or Bthe wrong product to be sent,^
they can in sum be very costly. Moreover, good-quality data
do not necessarily guarantee good decisions (Shah et al.
2012); nonetheless, low-quality data are usually associated
with poor decisions and high costs (Ballou and Tayi 1999;
Even and Shankaranarayanan 2007; Fisher et al. 2003). This
has been observed in several studies; for example, a study
conducted by the Data Warehouse Institute revealed that in
67% of the involved companies, poor DQ resulted in high
388
costs (Russom 2006). Moreover, 75% of the interviewees in
an international study on DQ admitted making wrong deci-
sions due to incorrect data (Harris Interactive 2006). In line
with these results, 85% of the interviewees in a study conduct-
ed by Forrester Research (2011) consider investments in DQ
and solving of DQ problems as important. Hence, managing
DQ is a significant problem for companies (Ballou et al. 1998;
Jiang et al. 2007; Russom 2006). Furthermore, ensuring ade-
quate DQ requires economics-oriented data quality manage-
ment (DQM) (Even and Kaiser 2009).
DQ is described as the extent to which the data stored in an
information system corresponds with the defined requirements
or with the respective information in the real world (Orr 1998;
Parssian et al. 2004). Moreover, DQ is considered to be multidi-
mensional (Wang and Strong 1996) with consistency, complete-
ness, and timeliness counted as the most cited among the DQ
dimensions (Lee et al. 2002; Wand and Wang 1996).
Researchers often focus on developing metrics to measure DQ
in terms of single DQ dimension (Ballou et al. 1998; Even and
Shankaranarayanan 2007; Heinrich et al. 2009; Heinrich and
Klier 2011; Hüner et al. 2011), which is necessary to quantify
as-is and to-be DQ levels and is a prerequisite to support
economics-oriented DQM (Heinrich et al. 2007b; Pipino et al.
2002). Besides, a large number of trade-offs and interrelation-
ships exist among the DQ dimensions (Ballou and Tayi 1999;
Ballou and Pazer 2003; De Amicis et al. 2006; Helfert et al.
2009). Consequently, enhancing a particular DQ dimension
(e.g., accuracy) using a specific DQI (e.g., data cleansing) will
most likely impact other DQ dimensions (e.g., timeliness)
(Ballou and Tayi 1999). Based on the results provided by the
aforementioned metrics and considering these interrelationships,
it is possible to select appropriate DQIs and determine an eco-
nomically reasonable execution sequence for them. The resulting
selection and its corresponding execution sequence constitute a
DQ improvement plan (DQIP), which can be applied to improve
DQ step-by-step in an economically reasonable manner.
However, an approach that considers interrelationships between
DQ dimensions in order to define an optimal selection and exe-
cution sequence of DQIs does not exist. Hence, we investigate
how an optimal DQIP can be determined, which consists of an
optimal selection and execution sequence of DQIs.
The paper is organized as follows. First, we outline the
problem context and discuss work related to determining an
optimal DQIP. Based on this discussion, we propose the deci-
sion model and a procedure for answering the aforementioned
research question in order to contribute to DQM. We illustrate
these explanations using examples with concrete DQIs and
DQ dimensions. Then, we evaluate the decision model with
respect to its quality by conducting a Monte Carlo simulation.
Next, we derive recommendations relevant for managing DQ
improvement initiatives as well as for further research. The
conclusion summarizes results, addresses limitations, and dis-
cusses areas of possible future research.
Kleindienst D.
Theoretical background and related work
Ensuring adequate DQ requires economics-oriented DQM
(Even and Kaiser 2009), so as to manage DQ with respect to
cost and economic benefits. An economical orientation can be
provided by considering performance indicators such as
Butility^ or Beffectiveness,^ and effort indicators such as
Bcosts.^ An economics-oriented DQM usually comprises of
four phases: defining, measuring, analyzing, and improving
DQ (Wang 1998). That is, DQ has to be improved continu-
ously using DQIs (Ballou and Tayi 1989; Wang 1998). DQIs
can be defined as actions that are taken to improve the actual
level of DQ (e.g., filtering data, buying trustworthy third-party
data, implementing consistency checks, etc.), which usually
results in costs and economic benefits (Heinrich et al. 2007a).
As DQ is a multidimensional concept, DQ dimensions and
their interrelationships have been analyzed in several studies.
For example, interrelationships between DQ dimensions such
as accuracy and timeliness as well as completeness and consis-
tency are modeled as trade-offs (Ballou and Pazer 1995; Ballou
and Pazer 2003). In addition, logical connections between DQ
dimensions have been discussed (Gackowski 2004) and ap-
proaches have been developed to quantify existing interrela-
tionships by means of correlations (De Amicis et al. 2006;
Lee et al. 2002). However, DQ dimensions are mostly analyzed
independently, because most frameworks for DQIs are devel-
oped on an ad-hoc basis to solve specific problems (Pipino et al.
2002). Nonetheless, taking into account these interrelation-
ships, which can be traced back to the differing effects of
DQIs on particular DQ dimensions, is a prerequisite for enhanc-
ing overall DQ (i.e., an aggregate of several DQ dimensions).
Against this background, it is also necessary to consider more
than one DQI when enhancing overall DQ, as single DQIs
mostly impact only a specific subset of DQ dimensions. As a
result, interrelationships between DQ dimensions influence the
optimal selection of several DQIs, as well as their optimal ex-
ecution sequence (Helfert et al. 2009) (cf. section 3). Thus, an
optimal DQIP comprises an economically optimal selection
and execution sequence of DQIs, considering the respective
interrelationships between relevant DQ dimensions.
Today, several concepts comparable to the idea of a DQIP
exist. In the following, we examine the following concepts:
allocating resources to different datasets with respect to costs
and effectiveness of available maintenance options (Ballou
and Tayi 1989), identifying the most valuable DQ enhance-
ment projects (Ballou and Tayi 1999), identifying an eco-
nomically optimal configuration of the dimensions com-
pleteness and accuracy (Even and Shankaranarayanan
2007), and setting priorities for DQIs based on an analysis
of the association between input and output accuracy
(Gelman 2010). Within the approach discussed by Ballou
and Tayi (1989), resources are not directly allocated to
DQIs, but to different datasets, depending on costs and the
Data quality improvement plan
effectiveness of available maintenance options. Although the
authors admit that Bin reality, of course, application of mul-
tiple procedures to the same dataset could be beneficial,^ it
is assumed that at the most one maintenance procedure can
be applied to a dataset. Besides this, Ballou and Tayi (1999)
propose an approach where the decision maker has to assess
the change in utility in terms of the current quality, required
quality, anticipated quality, priority of organizational activity,
cost of DQ enhancement, and added value for each project,
dataset, and relevant DQ dimension. Based on this
information, the most valuable DQ enhancement projects
can be identified. Apart from this, Even and
Shankaranarayanan (2007) model the costs and utility of
different configurations of DQ in data repositories in order
to identify an economically optimal configuration of the
dimensions completeness and accuracy. Based on this
configuration, the economic performance of data
repositories can be optimized. Additionally, Gelman (2010)
identifies the impact of input data errors in order to rank the
error effects. Input data with a higher negative effect in the
case of an error earn a higher priority in terms of DQI.
According to this prioritization, resources are allocated for
quality improvement of the respective datasets.
In a nutshell, solely Even and Shankaranarayanan (2007) and
Ballou and Tayi (1999) (indirectly) account for the multidimen-
sionality of DQ and the interrelationships between the DQ
dimensions; Even and Shankaranarayanan (2007) discuss the
case of only two dimensions, namely, completeness and accu-
racy. Moreover, none of the existing approaches provides in-
sights on an optimal execution sequence for the resulting
DQIs although this is necessary against the background of pos-
sible interrelationships. Consequently, we develop a formal de-
cision model that identifies the most effective DQIP by consid-
ering both selection and execution sequence of DQIs.
Fig. 1 Impacts on DQ
dimensions caused by DQIs
389
Determining an optimal DQIP
In this section, we describe the model’s main idea, introduce the
model’s definitions and assumptions, and develop the formal
decision model to determine an optimal DQIP. Finally, we pres-
ent an approach to solve the model by exhaustive enumeration.
Model idea
The aim of the following decision model is to consider inter-
relationships between the DQ dimensions and, based on these
interrelationships, identify the optimal selection and execution
sequence of DQIs. With regard to the interrelationships be-
tween DQ dimensions, the decision model examines the im-
pacts that the particular DQIs have on different DQ dimen-
sions, rather than measuring these interrelationships directly.
As illustrated in Fig. 1, the interrelationships between DQ
dimensions usually result from different impacts that the
DQIs have on several DQ dimensions. Thereby, a specific
DQI can have a positive impact on one or more DQ dimen-
sions, while having a negative impact on other DQ dimen-
sions simultaneously. For example, in the case of the three
DQ dimensions timeliness, completeness, and consistency,
there can be trade-offs in a certain situation. For example,
timeliness can be affected while attempting to achieve com-
pleteness and consistency; Bhaving complete or consistent da-
ta may need checks and activities, that require time, and thus
timeliness is negatively affected^ (Batini and Scannapieco
2006). Further, a trade-off can exist between consistency and
completeness; the more complete the data and the more data
there are, the greater the probability of inconsistencies among
these data. Thereby, DQIs designed to improve one DQ di-
mension would possibly cause a worsening of the other
(Batini and Scannapieco 2006). When it is known which types
390
of conflicting (or analogously complementary) impacts DQIs
can have on different DQ dimensions, we consider the inter-
relationships between the DQ dimensions where they arise.
Thus, we consider the interrelationships that result from
conducting DQIs that positively or negatively impact different
DQ dimensions at the same time. Therefore, the interrelation-
ships do not have to be determined explicitly, which is an
advantage as complex estimations of interrelationships be-
tween DQ dimensions are now obsolete.
In Fig. 1, the impacts of the three exemplary DQIs filtering
data (supposed to improve timeliness), buying additional ex-
ternal data (supposed to improve completeness), and consis-
tency checks (supposed to improve consistency) are shown as
tendencies using ↑ for positive, → for neutral, and ↓ for neg-
ative impact direction in accordance with Ballou and Pazer
(2003) as well as Batini and Scannapieco (2006).
To identify the optimal execution sequence of the DQIs, we
point to the next section, where an exemplary calculation
based on the mathematical impact modalities shows that the
DQI execution sequence matters.
As a result, DQIs have to be combined in an economically
reasonable manner, for example, based on costs and the effec-
tiveness of DQIPs as proposed in this paper. Therefore, an
available budget is allocated to the most effective DQIP com-
posed of a specific selection and execution sequence of DQIs.
We describe effectiveness of a DQIP as the extent to which the
gap between the recent and required DQ level is bridged by
applying a DQIP. In our context, bridging this gap by improv-
ing DQ implies augmenting the proportion of quality-assured
data in a dataset with respect to several DQ dimensions. In other
words, by applying a DQIP the data that are not quality-assured
(e.g., not timely, incomplete, inconsistent) are to be transformed
into quality-assured data (e.g., timely, complete, consistent).
Based on these ideas, we now develop a formal decision model.
Development of the decision model
In the following, we put forth the most effective DQIP from
the available DQIs within a given budget, while considering
the interrelationships between the DQ dimensions. The fol-
lowing definitions and assumptions form the basis of the
subsequent discussion.
[A.1] A DQ dimension di (with i = 1 , … , n) is impacted
by the application of DQIs vj (with j = 1 , … , m) with
given impacts I d i ;v j (with I d i ;v j ∈½−1; þ1 ). Therefore, im-
pact I d i ;v j ¼ −1 represents a maximum negative impact
and impact I d i ;v j ¼ þ1 represents a maximum positive
impact on a DQ dimension di.
[A.2] A DQIP comprises a selection out of m DQIs vj
(with j = 1 , … , m) and considers the execution se-
quence of the DQIs. Within a DQIP, all DQIs vj are
Kleindienst D.
applied sequentially and can each only be applied once
within the examined period. Therefore, a DQI vj is either
part of a DQIP or not.
[A.3] A DQIP is applied on one dataset with a fixed size
within one period. That is, a DQIP starts at the beginning
of a period and is completed at the end of the same period.
[A.4] The position of a DQI vj within an execution se-
quence is p (with p = 1 , … , m). Therefore, a DQI vj
applied on position p within a DQIP is written as vpj .
[A.5] Both initial and resulting DQ levels are expressed
as a proportion of quality-assured data in a dataset. The
DQ level for DQ dimension di, after applying a DQI vj on
position p within a DQIP, is Qpdi; v j (with Qpd i; v j ∈½0; 1 ).
The given initial DQ level of a DQ dimension di is Qpd i
(with p = 0 and Qpdi ∈½0; 1 ).
[A.6] A DQI vj causes fixed costs cj (with c j ∈Rþ ).
[A.7] In order to realize a DQIP, a budget B (with B∈Rþ )
is available. This budget is allocated to the most effective
DQIP, considering the selection and execution sequence
of DQIs.
[A.8] In the case of a negative impact (i.e., if I d i ;v j ∈½−1; 0½
), the DQ level on position p, Qpd i , results from a relative
reduction of the quality-assured part of the dataset on
position p − 1, Qdp−1 . In the case of a positive or no impact
i
I di ;v j (i.e., if I d i ;v j ∈½0; 1 ), the DQ level Qpdi results from a
relative increase of the part of the dataset that is not qual-
ity-assured, 1−Qdp−1 i
. The quality reduction or increase
p
caused by a DQI v j on a DQ dimension di follows from
multiplying impact I d i ;vpj with the respective (quality-as-
sured or not quality-assured) DQ level (Qdp−1 or 1−Qdp−1 ).
i i
This assumption is exemplarily visualized in Fig. 2.
As a consequence, the execution sequence of DQIs vj mat-
ters, because when a DQI is first applied with a positive im-
pact, it increases the quality-assured part of a dataset and
therefore offers a bigger target to a following negative impact,
Fig. 2 Calculation of positive and negative changes in DQ
Data quality improvement plan
thereby worsening the quality-assured data. On the contrary,
when a DQI is first applied with a negative impact, it reduces
the quality-assured part of a dataset, and therefore forms a
bigger basis for a subsequent positive impact to augment the
data that is not quality-assured. Hence, depending on the sign
of the impacts, different execution sequences lead to different
results, which is observed in Figures 3 and 4, where three
DQIs v1 (filtering data), v2 (additional external data), and v3
(consistency checks) with their respective impacts on three
DQ dimensions d1 (timeliness), d2 (completeness), and d3
(consistency) are applied in two exemplary sequences (v1 ➔
v2 ➔ v3 and v3 ➔ v2 ➔ v1). As it can be seen, the results of the
two sequences diverge. Thereby, even a small percentage dif-
ference can matter in the result in terms of monetary implica-
tions. In the given example, there is a difference of 20 percent-
age points in the results of the two sequences.
The position on which a DQI vj is placed within an execu-
tion sequence is expressed by the index p (with p = 1 , … ,
m), as defined in assumption A.4. Consequently, if consider-
ing an execution sequence of DQIs vj, and thus for impacts
I d i ;v j , the variables for the DQIs are written as vpj and I di ;vpj .
The optimization calculus determines which DQI vj finally
corresponds to which position p. The absolute effect of a
DQI vpj on a DQ dimension di follows from multiplying im-
pact I di ;vpj by the respective (quality-assured or not quality-
assured) DQ level (Qdp−1i
or 1−Qdp−1
i
).
In order to allocate a given budget in an economically
reasonable manner to an available set of DQIs, all possible
DQIPs need to be compared to each other. As a comparison
criterion, we calculate the respective effectiveness Ep = m of a
DQIP. That is, we calculate the extent to which the gap be-
tween the initial and the perfect DQ level is bridged by apply-
ing a DQIP, which is a vivid performance indicator in this
Fig. 3 Comparison of DQI execution sequences – sequence v1 ➔ v2 ➔ v3
391
context. In order to establish the effectiveness Ep = m of a
DQIP, the actual improvement has to be set in relation to the
highest possible improvement. The actual improvement is the
difference between the attained DQ level, ∑i¼1
n
wd i ⋅ Qm
d i , and
p¼0
the initial DQ level, ∑i¼1 wd i ⋅ Qd i . Similarly, the theoreti-
n
cally highest possible improvement is the difference between
the theoretically maximum DQ level, which is 1, and the ini-
p¼0
tial DQ level, ∑i¼1 wdi ⋅ Qdi . Hence, the objective function
n
maximizes the effectiveness of a DQIP in the following
manner:


p¼0
∑ni¼1 wdi ⋅ Qm
d i −Qd i
Maximize Ep¼m ¼ subject to∑mj¼1 cv j ⋅xv j ≤ B
1−∑ni¼1 wdi ⋅Qdp¼0
i
A detailed derivation of the objective function is given
in the appendix. To sum up, maximizing the effective-
ness Ep = m of a DQIP depends on the right selection of
DQIs vpj and right execution sequence, expressed in
terms of which vj corresponds to which vpj . In doing so,
a given budget is allocated on an available set of DQIs
and the most effective DQI execution sequence is iden-
tified while taking into account the interrelationships be-
tween the DQ dimensions.
Model solution
The objective function states a non-linear, binary opti-
mization problem with case distinction. An appropriate
methodology is required to identify the optimal im-
provement plan comprising both choice and sequence.
Consequently, in order to solve the decision model, we
considered several relevant combinatorial methods such
392
Fig. 4 Comparison of DQI execution sequences – sequence v3 ➔ v2 ➔ v1
as the traveling salesman problem or the knapsack prob-
lem. As none of these methods is capable of optimizing
both choice and sequence, especially with respect to an
unknown starting point, we argue that an enumeration is
the most appropriate method to solve the decision mod-
el and apply it in a three-step approach. In the first
step, all possible DQI selections are enumerated.
However, both the selections of DQIs and all possible
DQI execution sequences define a specific DQIP.
Therefore, knowing all DQI selections, we identify a
dominant execution sequence of the DQIs with respect
to their effectiveness in the second step. In the third
step, the DQIPs, including their dominant DQI execu-
tion sequences, are compared to each other. The
resulting decision is the most effective DQIP that keeps
within the budget.
The complexity of this enumeration rises exponentially
with the number of available DQIs. While a small number,
such as 4 DQIs, causes 65 calculations of the objective func-
tion, 10 DQIs cause approximately 10 million calculations,
and 20 DQIs cause approximately 6 × 1018 calculations of
the objective function. However, in real-world settings, since
a small number of possible DQIs is to be expected, enumera-
tion is a feasible method to solve the decision model at hand.
Moreover, the complexity can be minimized if the DQI selec-
tions that are not within the budget are excluded from the
outset. DQI selections can also be excluded if they are, ac-
cording to an expert, not technically feasible.
Evaluation on the basis of a monte Carlo simulation
In this section, we analyze the decision model with
respect to its quality (Hevner et al. 2004). Here, we test
the decision model’s quality in terms of its robustness
Kleindienst D.
by conducting an experimental evaluation based on a
Monte Carlo simulation. The Monte Carlo simulation
can be used to evaluate a deterministic model by using
pseudo-randomly generated numbers as parameter inputs
(Fishman 1996). For simulating a repeated sampling, a
large number of input parameter settings are created.
Based on these inputs, a deterministic computation is
performed and the results are aggregated to determine
the properties of the model’s behavior (Sawilowsky and
Fahoome 2002). This approach yields results within rea-
sonable time based on the law of large numbers and has
been used in a similar manner in previous research
(Fridgen and Müller 2011; Gelman 2012). With the
evaluation, we intend to show that the decision model
plausibly behaves to input parameter variations. In ad-
dition, we derive implications for the operationalization
of the decision model. Therefore, we first describe the
analysis procedure and provide a selection of visualiza-
tions appropriate for understanding the model’s behav-
ior. Then, a short summary of the key findings of the
analysis is given.
Simulation and analysis procedure
We use artificial data to simulate a repeated sampling and
pseudo-randomly create 1000 input parameter settings, as-
suming specific distributions as presented in Table 1.
Thereby, we choose an equal distribution for parameters de-
fined within an interval, and a Gaussian distribution for pa-
rameters where a deviation around a mean value is more real-
istic. In our case, this is reasonable as impacts can in fact take
values over their complete range of values and, in contrast,
costs vary around a certain mean with a corresponding vari-
ance. For the simulation, we consider three DQ dimensions di
and four DQIs vj. Hence, there are three initial DQ levels
Data quality improvement plan
Table 1 Input parameter settings
Range a
Distribution
Impacts, I d i ;v j [−1,+1] Equal
Initial DQ level, Qdp¼0
i
[0,1] Equal
DQI costs, cv j (μ,σ) = (1000, 20)3 Gaussian
Budget, B (μ,σ) = (3000, 60)3 Gaussian
Weights, wd i [0,1] Equal
a
When the costs cv j and the budget B had different orders of magnitude,
the decision model derived identical results
Qp¼0
d i , three weights wd i , four DQI costs cv j , and twelve im-
pacts I d i ;v j , altogether constituting one input parameter setting.
Based on each of the 1000 input parameter settings, we com-
pute the above developed decision model. That is, we deter-
mine each optimal DQIP and calculate its effectiveness Ep = m
and its costs ∑mj¼1 cv j ⋅ xv j .
In order to derive findings regarding the decision model’s
behavior based on the law of large numbers, we aggregate the
1000 results to a mean effectiveness (E ) and mean costs (C ).
In addition, we also vary other input parameters1 (i.e., impacts
I d i ;v j , budget B, DQI costs cv j , initial DQ level Qdp¼0
i
, and a
single impact I d i ;v j (e.g., I d1 ;v1 )) by +/−1%, +/−5%, +/−10%,
and +/−25%, ceteris paribus. In doing so, we study the deci-
sion model’s sensitivity and are able to compare the respective
behaviors of the decision model. As some of these different
parameter variations lead to similar implications and for rea-
sons of clarity, we present an appropriate selection of these
visualizations and derive feasible implications.
Visualization and analysis
In this section, we provide a discussion on chosen visualiza-
tions of the analysis results. As in sum the entire analysis is
grounded on more than 1.5 million executions of the simula-
tion, we identified specific visualizations to be appropriate for
understanding the model’s behavior. These visualizations are
coordinate systems showing the effectiveness E as a function
of either the initial DQ level Qdp¼0 i
or the exemplary single
impact I d 1 ;v1 .
In the illustrations in Figure 5, we show the behavior of the
mean effectiveness E as a function of the initial DQ level
Qp¼0
d i . In this manner, we can assess that the value of E is
descending with ascending values of Qp¼0 d i . The reason is that
less DQIs are coming into consideration to be part of the
optimal DQIP, the higher the initial DQ levels already are
(also see Figure 6). Additionally in Figure 5, we vary different
1
Weights, wd i , are not varied, as this would not provide adequate infor-
mative value.
393
input parameters, ceteris paribus. At first, it is obvious that
even a +/−25% variation in the budget B and costs cv j does
not cause a proportional change in mean effectiveness E.
Numerically, a +/−25% variation in both B and cv j causes
barely an average +/−1% deviation of E. Thus, it seems rea-
sonable to assume that an estimation error for budget and costs
would virtually cause no damage.
As argued above, the higher the initial DQ levels already
are, the less DQIs are coming into consideration to be part of
the optimal DQIP. This becomes apparent in Figure 6. The
reason therefore is that negative impacts have a stronger in-
fluence on DQ for higher initial DQ levels because there is
more potential to cause damage. For higher initial DQ levels,
DQIs that have negative impacts on one or more DQ dimen-
sions are therefore more likely to be sorted out by the decision
model. As Figure 6 takes an average view on Monte Carlo
simulation results, it shows non-integer values for the number
of DQIs.
In contrast and as illustrated in Figure 7, a variation in all
impacts I d i ;v j causes an approximately proportional change in
E, for example, a +/−10% variation in all impacts proportion-
ally causes an average +/−10% deviation2 in E. Similarly, a
+/−25% variation in all impacts causes an approximately pro-
portional deviation between +22% and −26%, respectively.
Therefore, an estimation error for impacts, rather than for bud-
get or costs, would lead to wrong results.
Additionally, a wrong estimation of impacts would be par-
ticularly problematic, as it would also implicitly be a wrong
estimation of the interrelationships between DQ dimensions.
Apart from that, in Figure 7 we see the behavior of the mean
effectiveness E as a function of one single impact, ceteris
paribus. Beside the similarly proportional deviation when
varying all other impacts, it is apparent that the DQI contain-
ing this single impact is most likely not a part of the optimal
DQIP as long as it is negative. The more positive this impact
becomes, the more likely is the application of the respective
DQI within the optimal DQIP and hence its benefit to the
optimal DQIP.
Results of the analysis and practical implications
The Monte Carlo simulations reveal the decision model’s
quality for DQM in terms of its robustness as well as implica-
tions for its operationalization. With regard to the decision
model’s robustness, in worst cases, we show that the devia-
tions in effectiveness and costs of the resulting DQIPs are
proportional to the strength of the input parameter variation.
In addition, we illustrate that the decision model’s behavior is
always as expected, technically reasonable, and comprehensi-
ble. That is, the model is being said to be robust.
2
This proportionality holds for a +/−1% or +/− 5% variation too.
394
Fig. 5 Variation of the input parameters budget and costs
The scientific key finding of the paper and its contribution
to DQ research is that the execution sequence of DQIs should
be considered in the approaches that improve DQ. Although,
literature is aware of the necessity to consider the interrela-
tionships between different DQ dimensions, it does not con-
sider the execution sequence of DQIs yet. Based on the anal-
ysis presented here, we can offer advice on certain
operationalization issues of the decision model:
(1) An estimation error for budget or costs is insubstantial in
terms of the effectiveness of the optimal DQIP. Therefore,
considering the chosen modeling technique as a basis,
operationalization efforts should rather focus on a realistic
estimation of the impacts, as they are the most sensitive
toward the effectiveness of the optimal DQIP. The
Fig. 6 Number of DQIs in a DQIP as a function of the initial DQ level
Kleindienst D.
importance of this issue is aggravated by the fact that an
estimation of the impacts is implicitly also an estimation of
the interrelationships between the DQ dimensions, which
are essential for realistic results. Moreover, an exact esti-
mation of positive impacts is more important than negative
impacts, as DQIs containing positive impacts are more
likely to be part of the optimal DQIP.
(2) Higher initial DQ levels imply that more DQIs will be
reasonably discarded by the decision model, as their appli-
cation would not lead to a substantial increase in effective-
ness. DQIs inherently contain positive and negative im-
pacts on different DQ dimensions. Lower initial DQ levels
imply that more DQIs containing negative impacts will be
considered for the optimal DQIP, as in such cases their
potential to cause damage is limited. However, as DQIs
with negative impacts can only contribute to an optimal
DQIP when applied together with DQIs having a positive
impact, it is therefore particularly important to consider an
economics-oriented DQI execution sequence.
Moreover, by providing the ability to identify an optimal
DQIP, the decision model can be useful for practitioners in
different ways:
(a) The DQIP reveals the most effective choice out of all
possible DQIs and thereby implicitly considers interde-
pendencies between different DQ dimensions. In this
manner, the DQIP helps practitioners in using a given
budget efficiently.
(b) Within the most effective choice of DQIs, the DQIP con-
siders path dependencies, that is, an optimal execution
sequence of the chosen DQIs can be identified. This
information is necessary for DQ project management.
(c) The example in this paper already enables practitioners to
learn about possible DQIs and interdependencies in the
Data quality improvement plan
Fig. 7 Variation of impacts
context of the most common DQ dimensions, namely
timeliness, completeness, and consistency. This forms
the basis for additionally applying the DQIP to other
context or company-specific DQ dimensions.
(d) With the help of the model leading to an optimal DQIP,
decision makers become aware of the complex problem
context at hand such as the multidimensionality of DQ or
the interrelationships between different DQ dimensions
and learn which information is necessary for their deci-
sions. In this manner, decision makers are better able to
make valid managerial decisions in the context of DQ.
Summary, limitations, and further research
In this paper, we developed a formal decision model that deter-
mines an economically optimal DQIP. Using this model, we
determine the best DQI selection and DQI execution sequence
with respect to the maximum effectiveness of the DQIP, keeping
within a given budget. In doing so, the decision model considers
interrelationships between the DQ dimensions in terms of com-
plementary and concurring impacts that DQIs usually have on
the different DQ dimensions. Thus, the model provides an inte-
grated view of different DQ dimensions. In order to demonstrate
the decision model’s quality in terms of robustness, we conduct-
ed an analysis based on a Monte Carlo simulation. Apart from
demonstrating the robustness of the decision model, we derived
practical implications for DQM. For example, a primary priority
is to accurately determine the impacts, and therefore implicitly
determine the interrelationships between the DQ dimensions.
For researchers, the paper depicts the so far unattended connec-
tion between existing interrelationships between DQ dimen-
sions and the necessity to consider a DQI execution sequence
in approaches to improve DQ.
395
Our findings are beset with some limitations particularly in
terms of simplifying assumptions. As applying a certain mea-
sure multiple times within an improvement plan could be pos-
sible in practice, confining the solution to scenarios where each
DQI is applied only once can be seen as a limitation. However,
we consider this assumption not to be too restrictive, as the
effect of applying one measure twice or even multiple times
can only have a marginal impact on the overall effectiveness
of an improvement plan due to the diminishing marginal bene-
fits. Hence, the model in its current version focusses on choos-
ing measures in terms of either applying or not applying them.
The process of adding an ability to choose the optimal number
of one certain measure within an improvement plan is lengthy,
but would lead to slightly more realistic results. Further research
could thus address the identification of data quality measures
that provide a higher positive impact if applied multiple times
within one time period. Another limitation of our model lies in
the assumption that the relative impacts of a certain data quality
measure are the same for different positions within an improve-
ment plan (e.g., updating data before or after applying another
improvement does not change the relative impact of the mea-
sure). Moreover, even if this assumption may not fully reflect
reality, we consider this assumption to be realistic to a high
extent, as positive impacts only affect the share of the data that
has not been subjected to quality assurance and negative impacts
only affect the remaining part of the data with regard to a certain
DQ dimension. Consequently, the shares that are affected by
impacts change and therefore the absolute effect of a DQ mea-
sure does depend on its position within a certain DQIP.
However, theoretically, there is no reason why the relative effect
of a measure to improve completeness, for instance, should
differ according to the number of data records to which it is
applied, i.e., it should be the same regardless whether it is ap-
plied to 10 incomplete data records or a million incomplete data
records. Finally, using synthetic data allows to tailor properties
396 Kleindienst D.

in order to meet various conditions that are not easily available Appendix
in reality (Barse et al. 2003), such as the used impacts. The use
of synthetic data in the Monte Carlo simulation is thus a limita- Supplementing the chapter BDevelopment of the Decision Model
tion of the paper, and an evaluation in a real-world context ^, a more detailed derivation of the objective function is
should be addressed in further research. Thus, these impacts given in the following.
could be determined in detail by conducting before-after com- The decision to consider a DQI vpj in a DQIP is described
parisons of the DQ dimensions when applying particular DQIs. formally using the decision variable xvpj , whereas the binary
Generally, several case studies should be conducted to further
variable xvpj is equal to 1 if DQI vpj is part of the DQIP, and 0 if
evaluate the decision model and its effects in real-world appli-
cations. However, we already derived recommendations that not. Thus, the resulting DQ level Qpdi; v j of DQ dimension di,
should be taken into account in DQM. Therefore, and despite after applying DQI vpj , is calculated in the following manner:
these pending tasks, we believe that the developed decision
8

model constitutes a sound contribution to DQM and an inspira- < 1−Qp−1 ⋅I
d i; v j d i ;vpj ⋅xvpj if I d i ;vpj ≥ 0
tion to researchers, as it particularly constitutes a first step to- Qpdi; v j ¼ Qdp−1 þ :
i; v j : Qp−1 ⋅I d ;vp ⋅xvp if I d ;vp < 0
ward considering the DQI execution sequence and interdepen- d i; v j i j j i j

dencies when determining an optimal DQI selection.

After remodelling and, for mathematical reasons, introduc-
Acknowledgments Supportive inputs and helpful comments by Dr. ing the substitute variables f di ;vpj and sd i ;vpj , the resulting DQ
Quirin Görz on an earlier version of this paper are gratefully
acknowledged. level Qpdi; v j can also be written as

( (
1−I di; vpj ⋅xvpj if I di; vpj ≥ 0 I di; vpj ⋅xvpj if I d i; vpj ≥ 0
Qpdi; v j ¼ Qdp−1 ⋅ f d i; vpj þ sd i; vpj ; with f di; vpj ¼ and sdi ;vpj ¼ :
i; v j 1 þ I d i; vpj ⋅xvpj if I d i; vpj < 0 0 if I di; vpj < 0

Based on this, the DQ level Qm

d i for dimension di, after
applying a complete DQIP for m DQIs vpj is calculated in the
following manner:

    
di ¼
Qm … Qdp¼0
i
⋅ f di ;v p¼1 þ s
di ;vp¼1 ⋅ f di ;vp¼2 þ s
di ;v p¼2 ⋅… ⋅ f di ;vp¼m−1 þ s
di ;v p¼m−1 ⋅ f p¼m þ s
di ;v j p¼m
di ;v j
j j j j j j

The binary variables xvpj , that are part of f di ;vpj and sdi ;vpj , In contrast to former approaches that have been crit-
neutralize the effect of DQI vpj
by taking the value 0 if a icized in literature for not considering interrelationships
p when aggregating DQ dimensions to an overall DQ lev-
specific DQI v j is not part of the DQIP. Although term (3)
el, in this decision model, interrelationships between the
contains all m DQIs vpj , a DQI selection is possible through
DQ dimensions d i are implicitly considered by the
this neutralization. Knowing the DQ level Qm d i for each DQ impacts I di ;vpj .
dimension di, the overall DQ level can be calculated. As there
Since all DQIs are applied on the same dataset, which has a
can be context-dependent differences between the DQ dimen-
fixed size, and a DQIP is realized within one period (cf. A.3),
sions, an allocation of weights to different DQ dimensions is
n
the costs cv j for applying DQI vj are fixed. As a result, the
reasonable. Therefore, we use a weight wdi (with ∑ wd i ¼ 1 ) overall DQIP costs are
i¼1
to weight the DQ dimensions in our decision model.

∑ni¼1 wdi ⋅Qm

di
∑mj¼1 cv j ⋅ xv j :
Data quality improvement plan
According to assumption A.8, the costs of a DQIP must not
exceed budget B; thus, the budget constraint for the decision
model is
∑mj¼1 cv j ⋅xv j ≤B
In order to allocate a given budget in an economically
reasonable manner to an available set of DQIs, all possible
DQIPs need to be compared to each other. As a comparison
criterion, we calculate the respective effectiveness Ep = m of a
DQIP in the way described in the chapter BDevelopment of
the Decision Model^. The objective function maximizes the
effectiveness of a DQIP in the following manner:


p¼0
∑i¼1
n
w d i ⋅ Qm
d i −Qd i
Maximize Ep¼m ¼ sub ject to ∑mj¼1 cv j ⋅xv j ≤B
1−∑ni¼1 wdi ⋅Qdp¼0
i
References
Ballou, D. P., & Pazer, H. L. (1995). Designing information systems to
optimize the accuracy-timeliness tradeoff. Information Systems
Research, 6(1), 51–72.
Ballou, D. P., & Pazer, H. L. (2003). Modeling completeness versus
consistency tradeoffs in information decision contexts. IEEE
Transactions on Knowledge and Data Engineering, 15(1), 240–243.
Ballou, D. P., & Tayi, G. K. (1989). Methodology for allocating resources
for data quality enhancement. Communications of the ACM, 32(3),
320–329.
Ballou, D. P., & Tayi, G. K. (1999). Enhancing data quality in data ware-
house environments. Communications of the ACM, 42(1), 73–78.
Ballou, D. P., Wang, R. Y., Pazer, H. L., & Tayi, G. K. (1998). Modeling
information manufacturing systems to determine information prod-
uct quality. Management Science, 44(4), 462–484.
Barse, E. L., Kvarnström, H., & Jonsson, E. (2003). Synthesizing test data
for fraud detection systems. Proceedings of the 19th Annual
Computer Security Applications Conference, Las Vegas, NV,
(USA). 384–395.
Batini, C., & Scannapieco, M. (2006). Data quality. Concepts, method-
ologies and techniques (data-centric systems and applications) (1st
ed.). Berlin: Springer.
De Amicis, F., Barone, D., & Batini, C. (2006). An analytical framework
to analyze dependencies among data quality dimensions.
Proceedings of the 11th International Conference on Information
Quality, Cambridge, MA, (USA). 369–383.
Even, A., & Kaiser, M. (2009). A framework for economics-driven as-
sessment of data quality decisions. Proceedings of the Fifteenth
Americas Conference on Information Systems. San Francisco,
California. Paper 436.
Even, A., & Shankaranarayanan, G. (2007). Utility-driven assessment of
data quality. The DATA BASE for Advances in Information Systems,
38(2), 75–93.
Fisher, C. W., Chengalur-Smith, I. N., & Ballou, D. P. (2003). The impact
of experience and time on the use of data quality information in
decision making. Information Systems Research, 14(2), 170–188.
Fishman, G. S. (1996). Monte Carlo; concepts, algorithms, and applica-
tions. New York [u.a.]: Springer.
397
Forrester Research. (2011). Trends in data quality and business process
alignment. Cambridge (USA).
Fridgen, G., & Müller, H. (2011). An approach for portfolio selection in
multi-vendor IT outsourcing. Proceedings of the 32nd International
Conference on Information Systems (ICIS), Shanghai, China.
Gackowski, Z. J. (2004). Logical interdependence of data/information
quality dimensions—A purpose-focused view on IQ. Proceedings
of the Ninth International Conference on Information Quality (ICIQ
2004), Cambridge, MA, (USA).
Gelman, I. A. (2010). Setting priorities for data accuracy improvements in
satisficing decision-makingscenarios: a guiding theory. Decision
Support Systems, 48(4), 507–520.
Gelman, I. A. (2012). A model of error propagation in conjunctive deci-
sions and its application to database quality management. Journal of
Database Management, 23(1), 103–126.
Harris Interactive. (2006). Information workers beware: Your business data
can't be trusted. Retrieved 10/13, 2008, from http://www.sap.com/
about/newsroom/businessobjects/20060625_005028.epx
Heinrich, B., Kaiser, M., & Klier, M. (2007a). How to measure data quality?
– a metric based approach. Proceedings of the 28th International
Conference on Information Systems (ICIS), Montreal, (Canada).
Heinrich, B., Kaiser, M., & Klier, M. (2007b). Metrics for measuring data
quality – foundations for an economic data quality management.
2nd International Conference on Software and Data Technologies
(ICSOFT), Barcelona, (Spain).
Heinrich, B., Kaiser, M., & Klier, M. (2009). A procedure to develop
metrics for currency and its application in CRM. ACM Journal of
Data and Information Quality, 1(1), 5:1–5:28.
Heinrich, B., & Klier, M. (2011). Assessing data currency — a probabi-
listic approach. Journal of Information Science, 37(1), 86–100.
Helfert, M., Foley, O., Ge, M., & Cappiello, C. (2009). Limitations of weight-
ed sum measures for information quality. San Francisco, CA, (USA).
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in
information systems research. Management Information Systems
Quarterly, 28(1), 75–106.
Hüner, K. H., Schierning, A., Otto, B., & Österle, H. (2011). Product data
quality in supply chains: the case of beiersdorf. Electronic Markets,
21, 141–154.
Jiang, Z., Sarkar, S., De, P., & Dey, D. (2007). A framework for recon-
ciling attribute values from multiple data sources. Management
Science, 53(12), 1946–1963.
Lee, Y. W., Strong, D. M., Kahn, B. K., & Wang, R. Y. (2002). AIMQ: a
methodology for information quality assessment. Information &
Management, 40(2), 133–146.
Orr, K. (1998). Data quality and systems theory. Communications of the
ACM, 41(2), 66–71.
Parssian, A., Sarkar, S., & Jacob, V. S. (2004). Assessing data quality for
information products: impact of selection, projection, and cartesian
product. Management Science, 50(7), 967–982.
Pipino, L., Lee, Y. W., & Wang, R. Y. (2002). Data quality assessment.
Communications of the ACM, 45(4), 211–218.
Provost, F., & Fawcett, T. (2013). Data science and its relationship to big
data and data-driven decision making. Big Data, 1(1), 51–59.
Radant, O., Colomo-Palacios, R., & Stantchev, V. (2014). Analysis of
reasons, implications and consequences of demographic change
for IT departments in times of scarcity of talent: a systematic review.
International Journal of Knowledge Management, 10(4), 1–15.
Redman, T. C. (2004). Data: An unfolding quality disaster. DM Review.
Russom, P. (2006). Taking data quality to the enterprise through data
governance. Seattle: The Data Warehousing Institute.
398
Sawilowsky, S., & Fahoome, G. C. (2002). Statistics through Monte
Carlo simulation with fortran. Rochester Hills: JMASM.
Shah, S., Horne, A., & Capellá, J. (2012). Good data won't guarantee
good decisions. Harvard Business Review, 90(4), 23–25.
Vera-Baquero, A., Colomo-Palacios, R., Stantchev, V., & Molloy, O.
(2015). Leveraging big-data for business process analytics. The
Learning Organization., 22(4), 215–228.
Kleindienst D.
Wand, Y., & Wang, R. Y. (1996). Anchoring data quality dimensions in
ontological foundations. Communications of the ACM, 39(11), 86–95.
Wang, R. Y. (1998). A product perspective on total data quality manage-
ment. Communications of the ACM, 41(2), 58–65.
Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: what data quality
means to data consumers. Journal of Management Information
Systems, 12(4), 5–33.