Advances in Engineering Software 34 (2003) 339–349

www.elsevier.com/locate/advengsoft

A multidimensional performance model for consolidating

Balanced Scorecards
Alain Abrana, Luigi Buglioneb,c,*
a
École de Technologie Supérieure (ETS), Université du Québec à Montréal, Montreal, Que., Canada
b
Software Engineering Management Research Laboratory (SEMRL), Université du Québec à Montréal, Montreal, Que., Canada
c
SchlumbergerSema, Via R. Morandi 32, I-00050 Rome, Italy
Received 1 July 2002; revised 17 January 2003; accepted 12 February 2003

Abstract
A Balanced Scorecard (BSC) presents the quantitative goals selected from multiple perspectives for implementing the organizational
strategy and vision. However, in most current BSC frameworks, including those developed for the Information and Communication
Technology field, each perspective is handled separately. None of these perspectives is integrated automatically into a consolidated view, and
so these frameworks do not tackle, either in relative or in absolute terms, the contribution of each goal to the whole BSC. Here, this issue is
highlighted, candidate consolidation techniques are reviewed and the preferred technique, the QEST model, is selected; more specifically,
three options are presented for incorporating the QEST model into a BSC framework.
q 2003 Published by Elsevier Science Ltd.
Keywords: Software performance measurement; Balanced Scorecard; QEST model; Indicators; Strategy

1. Introduction […], in improving organizational effectiveness in the

1.1. Context
Evaluating the IT function remains a challenge: well-
known financial measures such as ‘return on investment’
(ROI), ‘internal rate of return’, ‘net present value’ and
‘payback’ time have been demonstrated to be inadequate,
both in explaining IT investment decisions and in assessing
them. In assessing IT investments, it is crucial to understand
how organizational and strategic goals are achieved, and how
IT investments contribute to these goals. Since the financial
dimension alone is not sufficient and hides the relations (e.g.
cause and effect) among processes, an improvement step has
been suggested which would introduce additional dimen-
sions (or perspectives) of analysis [18,21]. Performance
management is defined by the United States Office of
Personnel Government as “the systematic process by which
an agency (organization) involves its employees (resources),
* Corresponding author. Address: Software Engineering Management
Research Laboratory (SEMRL), Université du Québec à Montréal,
Montreal, Que., Canada. Tel.: þ 39-33-8954-6917; fax: þ39-6-8307-4200.
E-mail addresses: luigi.buglione@computer.org; lbuglione@rome.
sema.slb.com (L. Buglione).
0965-9978/03/$ - see front matter q 2003 Published by Elsevier Science Ltd.
doi:10.1016/S0965-9978(03)00033-4
accomplishment of agency (organization) mission and
goals” [OPM—http://www.opm.gov/perform/overview.
htm]. The following five phases are defined for managing
organizational effectiveness:
† Planning work and setting expectations
† Continually monitoring performance
† Developing the capacity to perform
† Periodically rating performance in a summary fashion
† Rewarding good performance
1.2. Generic performance management and measurement
frameworks
Over the years, several frameworks have been developed
to address the management of organizational assets, both
tangible and intangible. The three main ones are the
Balanced Scorecard (BSC), the Intangible Asset Monitor
(IAM) and the Skandia Navigator.
1.2.1. Balanced Scorecard
In 1993, Robert S. Kaplan of the Harvard School
of Business, and consultant David Norton developed
340 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
Fig. 1. Balanced Scorecard: original perspectives [19].
the BSC, an evolution of the concepts in the Tableau de
Bord which emerged in France at the turn of the 20th
century [13]. The aim of the Tableau had been to translate
each company’s unitary vision and mission into a set of
objectives, through the identification of Key Success
Factors and Key Performance Indicators. Kaplan and
Norton defined the BSC [17] as a multidimensional
framework for describing, implementing and managing
strategy at all levels of an enterprise by linking, through a
logical structure, objectives, initiatives and measures to an
organization’s strategy. The resulting scorecard provides
the details of an enterprise view of an organization’s
overall performance: it complements the financial
measures with other Key Performance Indicators around
customer perspectives and internal business processes, and
around organizational growth, learning and innovation. It
must be noted that the BSC is not a static list of measures,
but rather a logical framework for implementing and
aligning complex programs of change, and, indeed, for
managing strategy-focused organizations. In summary, a
scorecard is to be used to facilitate the translation of
strategy into action.
The four basic perspectives of the original BSC are:
Financial, Customer, Internal Processes, Learning and
Growth (Fig. 1).
† Financial: this perspective typically relates to profit-
ability—it is measured, for instance, by the ROI, Return
On Capital Employed and Economic Value Added.
† Customer: this perspective includes several core or
generic measures of successful outcomes from the
company strategy, like, for instance, customer satisfac-
tion, customer retention, and market share in targeted
segments.
† Internal processes: this perspective focuses on the
internal processes that will have the greatest impact on
customer satisfaction and on achieving an organization’s
financial objectives.
† Learning and growth: this perspective identifies the
infrastructure the organization has to build and
manage to create long-term growth and improvement
through people, systems and organizational
procedures.
The execution of this strategy is then monitored
through an internal performance measurement framework
with a set of goals, drivers and indicators (both lag
and lead types) grouped into each of the four
perspectives.
The other key objective of a BSC is to tell the story of the
organization’s strategy. Three criteria help in determining
whether or not this objective has been achieved:
† Cause-and-effect relationship: every measure selected
should be part of a cause-and-effect relationship (causal
relationship chain) which represents the strategy.
† Performance drivers: the drivers of performance (lead
indicators) tend to be unique since they reflect what is
different about the strategy of a company.1
† Links to financial indicators: the various strategic goals
(such as quality, customer satisfaction and innovation)
must also be translated into measures which are
ultimately linked to financial measures.
1.2.2. Intangible Asset Monitor
Developed by Karl Sveiby, this framework [28,29]
classifies the intangible part of a balance sheet into a so-
called family of three thematic areas, each of them
composed from four categories (growth, renewal, efficiency,
stability/risk):
† Internal structure: consists of a wide range of concepts,
models, computers and administrative systems.
1
On the differences between BSC and the Goal–Question–Metric
(GQM) paradigm, refer to Ref. [6].
 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
Table 1
Comparison of performance and measurement frameworks
IAM BSC
Tangible Assets Tangible net book Financial Perspective
value
Intangible Assets Internal structure Internal Process
Perspective
External structure Customer Perspective
Individual competence Learning and Growth
Perspective
† External structure: relationships with customers and
suppliers, brand names, trademarks and image.
† Individual competence: people’s capacity to act in
different situations, including skill, education, experi-
ence, values and social skills.
The basic tenet of the IAM framework is that people are
the organization’s only profit generators, since every human
action is converted into both tangible and intangible
knowledge structures, in other words a ‘knowledge
perspective’.
1.2.3. Skandia Navigator
Developed by Edvinsson and Malone, this framework
[11,27] combines the two previous framework structures,
applying the BSC presentation format to the family of the
three IAM categorizations and using them under the
Intellectual Capital label in a supplement to the Skandia
Annual Report. Also, the Navigator uses the same four
perspectives (‘focus’) as the BSC,2 with a central and
pervasive ‘Human Focus’.
1.3. The Balanced Scorecard tailored for the ICT field
A few attempts to adapt the BSC for software intensive
organizations (SIOs3) have been reported, such as:
† the Balanced IT Scorecard (BITS) proposed by the
European Software Institute (ESI) [16,22 – 24]: provides
a new version of the four original perspectives (financial,
customer, internal process, infrastructure and innovation)
and adds a fifth one, the People Perspective.
† the BSC of Advanced Information Services Inc. [14]:
considers the ‘employee’ element as a distinct
2
The fourth perspective is called ‘Renewal and Development’ instead of
‘Learning and Growth’.
3
SIOs are organizations whose main objective is software development
and sales, departments or organizations which develop software as an
integral part of their end-products, and organizations which develop
software for internal use to achieve better business results or whose
software department can be described as an independent organizational unit
(European Software Institute definition, 1997).
341
Navigator ESI BITS AIS BSC
Shareholders’ equity Financial Perspective Financial
Organizational Capital Process Perspective Internal Process
Customer Capital Customer Perspective Customer
Human Capital Infrastructure and Learning and Growth
Innovation Perspective
People Employee
perspective, thereby expanding the analysis to five
perspectives (financial, customer, employee, internal
business process, learning and growth).
The five distinct perspectives derived from the original
scorecard and adapted to the ICT field are:
† Financial Perspective: How do our software processes
and software process improvement initiatives add value
to the organization?
† Customer Perspective: How do we know that our
customers (internal and external) are delighted with our
product?
† Process Perspective: Are our software development
processes performing at sufficiently high levels to meet
customer expectations?
† People Perspective: Do our people have the necessary
skills to perform their jobs and are they happy doing
so?
† Infrastructure and Innovation Perspective: Are process
improvement, technology and organizational infra-
structure issues being addressed with a view to
implementing a sustainable improvement program?
Table 1 summarizes how both tangible and intangible
concepts are being addressed in each of the generic
frameworks presented, as well as for those tailored to ICT
organizations.
1.4. An issue: determining the overall value in a BSC
Of these generic frameworks, Kaplan and Norton’s
BSC framework has the largest market penetration and
tackles performance at several levels, from the organiz-
ational level to the small business unit, and to the
individual level. However, while providing a vast array of
individual quantitative indicators, it does not provide for
consolidated performance values, either for the individual
perspectives or for their consolidation. Although in a BSC
the logical (i.e. causal) links are described textually, there
is no provision in a BSC for a quantitative consolidation
for each perspective to handle separately its various
342 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
numbers of indicators representing the various elements of
each business strategy. Even though it is structured
logically to represent the series of causal links between
the goals and drivers, a BSC framework does not provide
the support necessary to represent quantitatively how
much each perspective contributes, either in relative or in
absolute terms. In practice, the consolidation has to be
carried out intuitively by the users of the BSC; this
consolidation is, of course, subject to significant varia-
bility of assessment and interpretation depending on the
expertise and background of the users.
Current BSC frameworks function as ‘dashboards’, in
that they are made up of sets of individual indicators (like
the various gauges on an automobile dashboard) which
provide the elements necessary for monitoring the deploy-
ment of a (business) strategy. Knowing the business links
(causal relationships) across the various indicators, the
business executives must then, each time, figure out a
consolidated assessment of current organizational perform-
ance. Since BSC frameworks do not include techniques for
consolidating individual perspectives, this has to be done
subjectively, without the benefit of a formal quantitative
representation. Therefore, consolidated assessments which
can be communicated unambiguously and consistently to
those with a knowledge of the consolidation rules are not
possible.
Using a BSC, a manager can typically gain access to
detailed information about the goal, driver and indicator
(GDI) elements for each BSC perspective and about the
follow-up actions to take when results do not meet
established itemized thresholds. But, as with any
automobile dashboard or control panel, an overall
value of that perspective is not defined. It is therefore,
challenging to answer concerns at aggregate levels above
that of the details included in the BSC, whether for
internal or external purposes. This issue can be restated
in the following way: How can the consolidated
management view of performance, and the data collec-
tion view, be expressed? How, for instance, can a
question on process performance be tackled, a question
which, when translated into BSC terms corresponds to:
What is the overall value expressed by a set of indicators
chosen in your organization for the Internal Process
Perspective?
This paper addresses the consolidation issue and
illustrates the way in which a software performance
measurement model, namely the Quality factor þ
Economic, Social and Technical dimensions (QEST)
model, can be used for a BSC for information and
communication technology (ICT) organizations (referred
to as ‘ICT BSC’ in this paper).
1.5. Requirements and evaluation criteria
To tackle this issue, we have determined that the
following mandatory high-level capabilities have to be
added to the BSC framework and integrated into a
performance measurement system (PMS):
† A model providing a multi-perspective approach [15];
† A model which expresses these multiple viewpoints in a
single, consolidated value;
† Dimensional principle [8]: the dimensional principle
refers to the direct relationship between the number of
elements to be represented and the number of dimensions
used for their representation. A proper representation of
an entity must align the number of elements to be
represented and the number of dimensions used to
represent them. If this principle is not respected, the
danger is a loss of information. A well-known visual
example is the way in which the ancient Egyptians
painted human subjects. They used a 2D representational
style instead of a three-dimensional (3D) format. This
misalignment—translated from paintings to operational
data management—can cause similar misconceptions
when data is analyzed for the decision-making process.
† A model with a formal and consistent way to express
the outcomes.
In Section 2, candidate consolidation techniques are
identified and analyzed against the above evaluation
criteria. In Section 3, the QEST multidimensional perform-
ance measurement model is presented, with an emphasis on
its consolidation features; both the 3D original version and
its nD extension of the QEST model are presented. In
Section 4, three options are presented for combining the
QEST model in a BSC framework, thereby consolidating
the ICT BSC final outcomes into a single overall value. The
summary is presented in Section 5.
2. Candidate consolidation techniques
Some PMSs proposed in the literature are reviewed and
analyzed against these consolidation requirements.
† The Performance Pyramid [19] approach, by Lynch and
Cross, uses a pyramid-shaped ‘map’ for understanding
and defining the relevant objectives and measures for
each level of the business organization (four levels are
identified: business, core business processes, department
and teamwork, individuals).
† The Performance Prism [20], developed jointly by
Accenture and the Cranfield School of Management
(UK), allows two groups of stakeholders (investors
and customers) and up to five groups of stakeholders
(employees, suppliers, intermediaries, regulators and
communities) to be handled in the performance
management measurement scheme through a 3D shape
(a prism with 5 facets). This approach could be
positioned midway between a value chain like in the
EFQM Excellence Model [12] and the BSC. For
Table 2
Comparison of performance management—measurement models

Perspectives handled Model Dimensions Dimensional Performance Consolidated

type handled principle value for each performance
respected perspective value

Reference: Financial Closed N/A N/A No No

Balanced Scorecard—BSC Customer (Map)
(D. Kaplan and D. Norton) Internal process
Learning and growth
Candidate Performance Measurements Systems (PMS)

A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349

Performance pyramid Business Closed N/A N/A N/A N/A
(R. Lynch and K. Cross) Core business processes (Map)
Departments and workteams
Individuals

Performance prism Stakeholder satisfaction Closed G3 N No Yes

(A. Neely and C. Adams) Strategies
Processes
Capabilities
Stakeholder contribution
General framework for Can be used with any 4-perspective Closed N/A N/A Yes Yes
performance measurement set (typically the ones in the Kaplan and (Map)
(P. Rouse, M. Putterill and Norton’s BSC)
S. Ryan)

Table 3
Comparison of performance management—measurement models: the case for QEST models

Candidate PMS Perspectives handled Model type Dimensions Dimensional Performance value Consolidated
handled principle for each perspective performance value
respected

QEST 3D (Buglione and Economic, Open G3 Y Y Y

Abran) Social,
Technical
QEST nD (Buglione and QEST nD can be used with Open Gn Y Y Y

343
Abran) the desired set of perspectives
344 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
the purpose stated in this study, the following constraints
have been identified: the calculation matrix does not use
the geometry to derive the performance results, and it is
limited to handling five perspectives.
† The General Framework for Performance Measure-
ment [25], developed by Rouse, Putterill and Ryan at
the University of Auckland (NZ), uses Data Envelop-
ment Analysis models to represent performance
graphically with a 3D pyramid having a square base
where each of the four faces corresponds to the
original four BSC perspectives. Measures are linked to
the critical success factors and to the underlying
processes or cost drivers.
Table 2 presents a high-level comparison of these
three performance measurement approaches and the BSC
framework, including the following evaluation criteria:
the number of perspectives taken into account, whether
or not the approaches have a fixed structure or a dynamic
one allowing for a variable number of perspectives, the
representational format used and the provision of a
performance value for each perspective and a consoli-
dated value for all perspectives.
From Table 2, it can be observed that none of three
candidates respect at the same time the three criteria,
where a geometrical representation is used only to
present the distinct results visually.
Furthermore, no geometrical formula is associated with
the particular shape used for deriving performance indi-
cators, and the dimensional principle is not respected.
3. QEST: a multidimensional model for software
performance measurement
In addition to the above PMSs proposed for the generic
measurement of performance, a measurement model orig-
inally designed for measuring the quality and performance of
ICT projects, the QEST model, has been proposed by
Buglione and Abran. It is to be noted that this model is
generic and not restricted to the quality domain: it could be
generalized to the generic measurement of performance. We
will see that this QEST model does indeed meet all the
requirements and evaluation criteria for a consolidated
representation of performance. Note that some of the topics
discussed in this paper were developed individually and
presented, over a few years, in our previously published
papers. These papers were addressing some of the measure-
ment needs of middle managers somewhere in the organiz-
ational ladder. Two key characteristics of this new paper:
† this is the first time we have been able to combine these
individual pieces of research into a broader and integrated
organizational perspective;
† this paper also addresses some of the information needs of
managers with full organizational accountability.
3.1. The 3D basic model
The initial QEST model proposed a geometrical
representation of quality performance with the number of
dimensions corresponding to the number of viewpoints
considered; in addition, this number of dimensions can vary,
depending on the organization’s choice of the number of
dimensions they wish to take into consideration in their
PMS. The QEST model is thus not limited to 3 or 4
dimensions, but provides a multidimensional structured
shell which can then be populated with the criteria specified
for any project, on the basis of management objectives: it is
therefore, referred to as an open model. Such a topology in a
performance model also makes it possible to handle
multiple and distinct viewpoints, all of which can exist
concurrently in any software project. The various features of
the QEST model have been documented:
† Theoretical considerations [2,3];
† Geometrical and statistical foundations [4];
† Implementation of the model [10];
† Quality Factor [3] and Quality Factor through QFD [7].
The basic purpose of the structured shell of this open
model is to express performance as the combination of the
specific measures (or sets of measures) selected for each of
the specified dimensions, these values being derived from
both an instrument-based measurement of productivity and
a perception-based measurement of quality in the initial
QEST quality model. The QEST models handles both
weighting factors and thresholds; the selection of weights
and thresholds is of course context dependent.
For demonstration purposes, when three perspectives are
selected for assessing quality (and performance), a 3D
geometrical representation of a regular tetrahedron is
selected as the basis for the QEST model, and is illustrated
in Fig. 2:
† the three measurement dimensions ðE; S; TÞ in the spatial
representation correspond to the corners of the pyramid’s
base, and the convergence of the edges to the P vertex
Fig. 2. QEST model and its hyperplane sections corresponding to two
assessments at different points in time, relative to normalized targets.
 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
correspond to the maximum performance level—the
normalized target;
† when the three sides are of equal length, the solid shape
that represents this 3D concept is a pyramid with its
triangular base and sides of equal length (tetrahedron).
The use of this pyramid-type geometrical representation
imposes the following constraint: the sides must be equal. In
practice, this constraint is met by normalizing the target
values to 1 for each of the three different dimensions
chosen—and with the representation of corresponding sides
with a length value of 1 (regular tetrahedron). This allows
the representation of the dimensions through a normalized
set of values between 0 and 1 for each axis and on a ratio
scale.
With this 3D representation, it is possible to determine
and represent performance using the three classic geometri-
cal concepts: distance, area (the hyperplanes in Fig. 2) and
volume; furthermore, the ratio between the volume of the
lower part of the truncated tetrahedron and the total volume
of the tetrahedron itself represents the normalized perform-
ance level of a project being assessed against its
performance target which is specified as the apex of the
tetrahedron, as demonstrated in Ref. [4].
Thus, the volume within this 3D shape immediately
shows in a visual way the assessment of quality at a point in
time against the specified target represented, in a normalized
way, by the apex of the tetrahedron; similarly, when
performance is reassessed throughout a project life cycle,
the sloped section of the hyperplane represents how
performance is progressing in the three dimensions, and
whether or not quality is progressing within each dimension.
This is the added visual value that a geometrical
representation can provide, together with the support of
the corresponding geometrical formula, to adequately
consolidate the quantitative values of the concepts measured
individually for each edge. By analogy, a management
dashboard’s power is limited to a series of individual gauges
and controls providing a feel for the distinct elements; it
does not provide for a combined representation or for the
overall status of a given phenomenon. To summarize, the
geometrical approach of the QEST model makes it possible
to represent the measurement of performance in a simple
and visual way for an immediate grasp of the current status
of project performance at any time.
Furthermore, an extension of the QEST model to the
whole software life cycle has been developed, namely the
LIfecycle MEasurement (LIME) model [5]; this QEST/
LIME model is the QEST instantiation to the whole
software development life cycle, and is applied to each
development phase.
3.2. The n-dimensional extension
Of course, other profiles could be included in a
QEST/LIME model, including quality models such as
345
EFQM or the Malcolm Baldridge models, or others like
the BSC or the Skandia Navigator where the number of
stakeholder viewpoints can be significantly increased.4
While they have been described and illustrated using
three dimensions, the QEST/LIME models, being open
models, are not restricted to the representation of three
dimensions of performance. Their geometrical approach is
generic and allows for the handling of any number of
dimensions, through the use of appropriate algebra corre-
sponding to the number of dimensions selected, that is, the
use of n dimensions. This was demonstrated in Ref. [9]: the
mechanism to be used for n dimensions is a simplex,
definable as the convex hull of any d þ 1 affinely
independent points in some Rn ðn . dÞ; where d represents
the number of dimensions taken into account. There are, of
course, both advantages and disadvantages to this. For
instance, by using matrix calculations, it is possible to
extend the underlying concepts of the QEST/LIME models
to n dimensions, and we have demonstrated the backward–
forward validity of this mechanism for any number of
dimensions. The models are thus able to take into account a
greater number of stakeholders. However, from the fourth
dimension on, it is no longer possible to take advantage of
the visualization of related concepts in a direct way.
In Table 3, a summary review of the characteristics of
both the QEST 3D and QEST nD models are presented
using the same criteria as those listed in Table 2: both QEST
models respect the dimensional principle and allow for
consolidation for each individual perspective, as well as for
consolidation of all perspectives considered. Table 3
highlights the relevance of investigating the use of the
QEST models, in particular QEST nD, the n-dimensional
extension and generalization of the basic model, to a generic
ICT BSC.
4. Integration of QEST and BSC
The two options for integrating QEST and BSC are
discussed next. For simplicity’s sake, in this text examples
are limited to the specific case of n ¼ 3:
1. BSC (QEST nD): every ICT BSC perspective applies the
basic QEST model as a measurement system using
4
Atkinsons, Waterhouse and Wells [1] distinguish between two groups
of stakeholders: Environmental Stakeholders, who belong to the external
environment of the organization, which defines the critical elements of its
competitive strategy (customers, owners, community); Process
stakeholders, who are employees and suppliers working within the
environment defined by the external stakeholders to plan, design,
implement and operate the processes that make the company’s product
and deliver it to its customers. In addition, Sharp, Finkelstein and Galal [26]
propose a 5-step procedure for the identification of stakeholders in the
requirement engineering process, where there are four baseline
stakeholders: users (mapping to the social perspective), developers
(mapping to the technical perspective), decision-makers (mapping to the
economic perspective) and legislators.
346 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
the related nD extension (i.e. QEST 5D for 5 perspec-
tives, and so on);
2. QEST nD (BSC perspective): each ICT BSC perspective
represents one of the dimensions of the tetrahedron
(taking into account the n-dimensional extension of
QEST).
4.1. First option: BSC(QEST nD)
Here, the BSC would manage separate, and often
disjointed, performance values. This would produce n
different quantitative outcomes for each perspective, with
only descriptive linkages at the strategy level, and no clear
traceability of quantitative contributions to the overall
strategy level.
The mapping of the elements of the BSC and QEST
frameworks is presented in Table 4.
Some differences can be noted between the two frame-
works. First of all, what in QEST is referred to as a
perspective would correspond to the logical sum of a goal
and one of its related drivers in a BSC perspective, a more
granular subdivision than in QEST. Thus, this group of
elements (goal þ its related drivers) is directly referred to
a list of n indicators. In the same way, in the QEST
model the indicators are directly referred to a perspective
(i.e. economic, social or technical). Another difference is
the ability of the QEST model to handle normalized values;
this is also, of course, a constraint.
Since QEST handles normalized values, some
additional BSC framework elements must be gathered:
lower thresholds for each measure in order to derive a
normalized value for each goal þ driver (G þ D) topic
and its weight, where the sum of the G þ D perform-
ances for each QEST perspective will be equal to 1; in
Fig. 3, G1D2 stands for goal no.1 – driver no.2 within a
certain BSC perspective.
Extending the concept to n possible tuples of GxDy
elements, the application of the QEST nD calculation
mechanism will produce two ðn þ 1Þ square matrices, upper
measure (UM) and grand measure (GM) from which the p
values (the ratio of UM to GM) will be derived. The final
Table 4
Mapping of BSC and QEST structural elements
BSC QEST
Perspective Perspective
Goal
Driver
Indicator Indicator
Measure Value collected in the field
– Weight
Target Upper Threshold
– Lower Threshold
– Normalized value
Initiatives SPI-related process area for
improvement
Fig. 3. QEST application to a single BSC perspective.
outcome will be a normalized measure indicating the
performance value for a specific perspective. Repeating this
calculation for all the possible perspectives taken into
account, the ICT BSC (QEST nD) will return a ‘first-level
value’5 for internal benchmarking analysis. This value can
be useful whether or not software-intensive organizations
intend to maintain the same GDI (goal –driver –indicator)
architecture during (at least) two subsequent time sequences
in order to monitor its evolution over time.
We illustrate in Fig. 4 how, for a given BSC perspective
(e.g. Infrastructure and Innovation), the QEST approach can
be applied. The consolidated normalized performance value,
following QEST application, would be equal to 0.9808.
4.2. Second option: QEST nD(BSC)
Here, with the joint application of QEST and BSC, the
QEST model represents the ‘glue’ for the whole BSC
framework and provides a unique consolidated value for all
the perspectives considered.
This second option addresses the following concern:
after considering the internal performance value for each
perspective, the next logical question an executive asks
when assessing the overall business view is: What is the
consolidated view of the whole BSC value? That is, is it
possible to obtain an overall performance value first, and
then carry out a drill-down in-depth analysis (e.g. top –down
analysis) from there, perspective by perspective, looking for
single indicators and relationships hypothesized across
goals selected in the initial design of the organizational
strategic BSC map? In this case, the mapping between
QEST and BSC is easier, as shown in Table 5.
The concept of ‘perspective’ is the same in this case,
while the value for each of the perspectives taken into
account (Fig. 5) is that derived from the calculations shown
in Fig. 4. This means that each perspective will have only
one performance value from the BSC(QEST nD) calcu-
lation, proposed in Section 4.1. In this case, the QEST
5
In this paper, the term ‘first-level value’ is used to express the
consolidated value from a tree of values, as a result of the QEST
calculation algorithm.
 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349 347

Fig. 4. QEST application to a single BSC perspective: map and values.

Table 5 model must take into account just the three values as
Mapping of BSC perspectives and QEST structural elements they are.
BSC QEST
Extending the concept to n possible perspectives, the
application of the QEST nD calculation mechanism will
Perspective Perspective produce two ðn þ 1Þ square matrices, UM and GM, the p
Perspective value from QEST Indicator values being their ratio. The final outcome will be a
application normalized measure representing the consolidated value of
Measure Value collected in the
the whole BSC. Repeating this calculation at regular time
field
– Weight intervals (either projects or business intervals), a ‘con-
Target Upper Threshold solidated top-level value’ for internal benchmarking
– Lower Threshold analysis is available to management, provided, of course,
– Normalized value that the organization maintains the same GDI architecture
Initiatives SPI-related process area for
across successive time intervals.
improvement

4.3. Integration of the options: BSC(QEST nD) þ QEST

nD(BSC)

From the two previous sub-sections, it can be observed

Fig. 5. QEST application to the whole BSC framework.
that these two ‘options’ are not mutually exclusive, but can
be seen as complementary to one another. An implemen-
tation sequence is shown in Fig. 6.
The first phase consists in the classic construction of the
BSC strategic map, which describes the causal links among
the organization’s goals. The next phase consists in its
‘translation’ into a set of consolidated performance values,
one for each perspective (through the ‘QEST nD[BSC
perspective]’ option). The output will constitute five
performance values, which will represent the inputs for

Fig. 6. Sequence for integrating the QEST model into the ICT BSC framework.
348 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
the overall BSC calculation (Phase 3). Here also, the
application of the QEST nD calculation mechanism,
through the use of the simplex, will produce a ð6 £ 6Þ
matrix. Its result will be the final consolidated performance
value p for the whole BSC.
5. Conclusions
The BSC framework provides senior executives with a
powerful analytical tool for putting their organizational
strategy into action. Causal relationships are stressed in the
BSC structure, and this makes it possible to aggressively
pursue, through close monitoring, the detailed actionable
items supporting the corporate strategy. However, the BSC
framework does not formally specify ‘how’ a given BSC
perspective (through its GDI elements) contributes to the
whole BSC value, nor does it provide for a consolidated
performance value for the whole organization, and the
global strategic target.
The availability of such consolidation rules would
provide executives with single values as well as the
corresponding information model for interpreting the
information provided. These rules would also allow for a
top-down decomposition, so that an understanding could be
gained of the background forces driving each consolidated
view. The availability of consolidated values could, in turn,
lead to the establishment of standard sets of consolidated
measures and to the institutionalization of internal –external
benchmarking practices.
Key concepts of the original Kaplan and Norton BSC
framework were presented, together with highlights of the
tailoring available for the ICT field. Some of the value of
BSC is in the separate treatment of factors. However, such
separate treatment has also some limitations: for instance,
there is in the BSC approach an hypothesis that, when
managers define the various views, the linkages across
views are both properly described, properly implemented
and that the strategic linkages are indeed present. However,
the current BSC technology does not provide techniques to
formally define, verify, implement, consolidate and analyze
such linkages. Some candidate PMSs for improving a BSC
framework were identified and analyzed against a list of
relevant criteria, pointing to some of their limitations in this
context of intended use. The QEST model was presented
next and discussed with respect to the same set of evaluation
criteria. While the basic initial QEST model can be used for
three perspectives at once, its extension and generaliz-
ation—called QEST nD—makes it possible to handle any
number of perspectives simultaneously, including their
multidimensional consolidation. In fact, senior managers
needs both to make high level synthesis of information
coming from various partial views and, simultaneously, the
ability to breakdown the information into much more
granular view to get more explanatory power. That is, the
managers (and in particular senior managers) need to both
roll-up and roll-down the pieces of information. The current
technology of BSC does not allow for full roll-up of the
information. This is a key strength of QEST nD which, in
addition, does it with known geometrical concepts, as well
as with multiple views at once. Of course, it does not ignore
the roll-down features of BSC, nor the somewhat limited
roll-up features but it rather enhances these roll-up features.
Options for integrating the BSC and the QEST nD model
were presented next. The first option consisted in managing
the BSC perspectives separately, while the second one
consisted in consolidating the perspective performance
values to derive the overall BSC value. Both options are
valuable and valid for capturing different measurement
viewpoints: the partial and overall performance values
expressed by an ICT BSC system. It is possible to read data
and analyze them with a ‘layer’ logic, from top to bottom,
considering each BSC perspective as a separate layer
contributing to the final result.
The three levels of analysis a manager should perform
could therefore, be:
1. Look at the overall BSC value (the ‘BSC (QEST nD)’
option);
2. Look at the single perspective values (the ‘QEST
nD(BSC)’ option);
3. Analyze each single perspective for follow-up action on
the basis of the usual BSC ‘strategic map’, looking at
results in the single ‘layer’ as well as the contribution
resulting from the causal links across the goals from the
different perspectives.
In this way, QEST nD becomes a n-dimensional
performances measurement system providing a consoli-
dated ‘value’ to the ICT BSC and to the proxies it takes into
account, with a holistic focus on the whole BSC.
Consolidated numbers focuses the attention, and (through
the related synthesis—done with pre-defined and agreed
upon rules) facilitate the communications of strong
messages. Of course consolidated figures do not provide
cause and effect relationship—this is provided by the roll-
down features and facilities.
The research work presented here has put the emphasis
on the measurement aspect in performance management.
Undoubtedly, case studies and empirical evidence are useful
in the context of technology transfer (e.g. in terms of
grasping the concepts conveyed by the model, and to present
hints and tips to implement them in terms of either success
or failure factors). Right now, neither case studies, or
empirical evidence, demonstrate the validity of such model,
since its novelty, waiting for a trial phase during next
months in ICT companies, mainly already adopting BSCs,
interested in this approach.
Last but not least, further investigation is required to
tackle more specialized issues, such as the following: Is it
possible to measure and ‘weight’ the contribution in terms
of performance that goals from one perspective provide to
 A. Abran, L. Buglione / Advances in Engineering Software 34 (2003) 339–349
another, with a view to capturing the inner value of each
perspective in the value chain, and how much is contributed
by the other perspectives?
References
[1] Atkinson AA, Waterhouse JH, Wells RB. A stakeholder approach to
strategic performance measurement. Sloan Mgmt Rev 1997;Spring:
25–37.
[2] Buglione L, Abran A. A quality factor for software. Proceedings of
QUALITA99, Third International Conference on Quality and
Reliability, Paris, France; March 25 and 26, 1999. p. 335–44
[3] Buglione L, Abran A. Multidimensional software performance
measurement models: a tetrahedron-based design. Rapport de
Recherche No. 99/01. Département d’informatique, UQAM, Uni-
versité du Québec à Montréal; 14 Mai 1999
[4] Buglione L, Abran A. Geometrical and statistical foundations of a
three-dimensional model of performance. Int J Adv Engng Software
1999;30(12):913–9.
[5] Buglione L, Abran A. LIME: a three-dimensional software perform-
ance measurement model for project management. Proceedings of
the Second World Congress on Software Quality (2WCSQ), Yoko-
hama (Tokyo Bay Area); September 25– 29, 2000
[6] Buglione L, Abran A. Balanced Scorecards and GQM: what are the
differences? FESMA/AEMES Conference, Madrid (Spain); October
18–20, 2000
[7] Buglione L, Abran A. QF2D: quality factor through QFD application.
Proceedings of Qualita2001 (Fourth International Congress on
Quality and Reliability), Annecy, France; March 22 and 23, 2001.
p. 34–9. ISBN 2-9516453-0-0
[8] Buglione L, Abran A. Multidimensionality in software performance
measurement: the QEST/LIME models. Proceedings of SSGRR
(Second International Conference on Advances in Infrastructure for
Electronic Business, Science, and Education on the Internet),
L’Aquila (Italy); August 6–10, 2001
[9] Buglione L, Abran A. QEST nD: n-dimensional extension and
generalisation of a software performance measurement model. Int J
Adv Engng Software 2002;33(1):1– 7.
[10] Buglione L, Abran A. Implementation of a three-dimensional
software performance measurement model. Technical Report. Uni-
versité du Québec à Montréal; 2003
[11] Edvinsson L, Malone T. Intellectual Capital: realising your compa-
ny’s time value by finding its hidden brainpower. New York: Harper
Collins; 1997.
[12] European Foundation for Quality Management. The EFQM excel-
lence model—improved model. EFQM; 1999
[13] Epstein MJ, Manzoni JF. The Balanced Scorecard and Tableau de
Bord: a global perspective on translating strategy into action.
INSEAD Working Paper 97/63/AC/SM; 1997
349
[14] Ferguson P, Leman G, Perini P, Renner S, Seshagiri G. Software
process improvement works! SEI Technical Report, CMU/SEI-TR-
99-27; November 1999
[15] Finkelstein A, Sommerville I. The viewpoints FAQ. Software Engng J
1996;11(1):2 –4.
[16] Ibáñez M. Balanced IT Scorecard Generic Model Version 1.0.
Technical Report. European Software Institute, ESI-1998-TR-009;
May 1998
[17] Kaplan RS, Norton DP. The Balanced Scorecard: translating strategy
into action. Harvard Business School Press; 1996.
[18] Land FF. Evaluation in a socio-technical context. Organizational and
social perspectives on information technology. Baskerville R, Stage J,
DeGross JI, editors. IFIP TC8 WG8.2 International Working
Conference on the Social and Organizational Perspective on Research
and Practice in Information Technology, Aalborg, Denmark; 2000,
chapter 8
[19] Lynch R, Cross K. Measure up!, 2nd ed. Blackwell Publications;
1995. ISBN 1557867186.
[20] Neely A, Adams C. Perspectives on performance: the performance
prism. Center for Business Performance (CBP), Cranfield University,
School of Management, URL: http://www.som.cranfield.ac.uk/som/
cbp/prismarticle.pdf
[21] Opdhal AL. A comparison of four families of multi-perspective
problem analysis method. Proceedings of IRIS 20 Conference, Hanko
(Norway); August 9–12, 1997
[22] Reo D. Applying the Balanced Scorecard for process improvement—a
case study by the European software industry. Presentation,
Measuring The Business Value Of IT With The Balanced Scorecard,
Phoenix, AZ; February 23 and 24, 2000
[23] Reo D, Quintano N, Ibáñez M. ESI Balanced IT Scorecard Process
Perspective V 1.1. Technical Report. European Software Institute,
ESI-1999-TR-016; February 1999
[24] Reo D, Quintano N, Buglione L. ESI Balanced IT Scorecard
Infrastructure & Innovation Perspective Technical Report. European
Software Institute, ESI-1999-TR-043; December 1999
[25] Rouse P, Putterill M, Ryan D. Towards a general managerial
framework for performance measurement: a comprehensive highway
maintenance application. J Prod Anal 1997;8(2):127–49.
[26] Sharp H, Finkelstein A, Galal G. Stakeholder identification in the
requirements engineering process. Proceedings of DEXA 99 Con-
ference, 10th International Workshop on Database and Expert
Systems Applications, September 1–3, 1999, Florence, Italy. IEEE
Computer Society Press; 1999. p. 387–91
[27] Skandia Group. Skandia UNIC: Universal Networking Intellectual
Capital; 1998, URL: http://www.skandia.se/group/fininfo/frame_
annual_reports.htm
[28] Sveiby KE. The new organisational wealth: managing and measuring
knowledge-based assets. San Francisco: Berrett-Koelher; 1997.
[29] Sveiby KE. The Intangible Asset Monitor; 1998. URL: http://www.
sveiby.com.au/IntangAss/CompanyMonitor.html