Developing RAE1

See
discussions, stats, and author profiles for this publication at: http://www.researchgate.net/publication/256106276
Developing a Combined Framework for

Evaluating IT Projects based on IT-BSC and
COBIT
ARTICLE in INTERNATIONAL JOURNAL OF DIGITAL CONTENT TECHNOLOGY AND ITS APPLICATIONS MAY 2011
DOWNLOADS
VIEWS
191
67
1 AUTHOR:
Reza Samizadeh
Alzahra University
11 PUBLICATIONS 7 CITATIONS
SEE PROFILE
Available from: Reza Samizadeh

Retrieved on: 18 August 2015
Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
Developing a Combined Framework for Evaluating IT Projects based on

IT-BSC and COBIT
1
Nasrin Dastranj Mamaghani, 2 Reza Samizadeh, 3 Fatemeh Saghafi

1, Corresponding Author
Iran Telecommunication Research Center, dastranj_n@itrc.ac.ir
2,
Department of Science and Technology, Alzahra University, rsamizadeh@alzahra.ac.ir
3,
Iran Telecommunication Research Center, saghafi@itrc.ac.ir
doi : 10.4156/jdcta.vol5.issue5.2
ABSTRACT
IT projects have special features like high risks, intangible outcomes, hidden benefits and high costs
that make the use of traditional evaluation techniques very difficult and the outcomes unreliable,
therefore information technology (IT) investments should be carefully controlled and evaluated. Large
efforts have been done to provide appropriate methods for evaluating IT projects. In this paper,
traditional methods of measuring projects are reviewed and advantages and disadvantaged of them in
measuring IT projects are specified. Then new techniques that are specialized in measuring IT projects
are introduced and IT-BSC as an appropriate method with available information is selected. As COBIT
framework is defined for IT projects, from the combination of COBIT and IT-BSC, a framework for
evaluating IT projects is proposed. Besides, the proposed framework is validated by experts opinion
with questionnaire and appropriate hypothesis test. Finally, an implemented project in IT area is
selected as a case study and is evaluated by measures of the proposed framework. The results show
that this framework can be very effective in evaluating IT projects.
Keywords: IT-BSC, COBIT, Delphi Technique, IT Project, Project Management

1. Introduction
In today competitive business environment, there is strong need to control costs and returns on
investment along with lower risk of capital investment. Understanding the potential impact of IT on
strategic power of organizations and increasing payments of IT projects, has made the evaluation and
control of capital investments in IT projects remarkably important and critical [1,3]. According to
studies, only 18% of organizations use proper methods of calculating benefits of capital investment in IT
[1] and at least 22% of capital investment in IT is regarded as waste. Also, 34 to 40 percent of IT
projects have no benefit to be obtained [4.5]. Reasons for this failure are complex. Technical factors, and
in addition to that, environmental, organizational, managerial and human resources contribute to this
failure. In some studies [1, 2, 3] lack of correct evaluation and control of IT projects is the main factor of
failure in this projects. In this paper, different methods of evaluating capital investment of IT projects are
classified. In Section 2 the traditional assessment methods of projects are investigated and their
advantages and disadvantages for evaluating IT project are expressed. Section 3 introduces new ways to
evaluate IT projects. They are specific methods of IT projects assessment. Among these methods ITBSC 1 and COBIT 2 are selected as they are specified for IT projects and their information can be
available. With merging these two frameworks, a framework for evaluating organizational IT projects is
presented in Section 4. Finally, a sample project is evaluated using the proposed framework.
2. Traditional methods of project evaluaiont

Many studies investigated the benefits and drawbacks of traditional methods. Main goal of the
organization is to increase profits and wealth of project stakeholders. To calculate profit and loss of a
project, operational methods of assessment are needed like traditional methods. Also these methods are
known and understandable and are based on accepted principles [9,13,14]. But there are a lot of
1
2
Balanced Scorecard
Control Objectives for Information and related Technology
- 10 -
disadvantages too. Disadvantages of using traditional methods in evaluating IT projects can be

categorized in two groups: conceptual reasons and operational reasons.
Conceptual reasons: When a company invests in the IT project, five different partners will be
involved: Main Organization (project owner), users, project teams, sponsors and shareholders. Each of
the partners follows different objectives to meet outputs of the project. If mentioned methods are used,
only the goal of project manager that is the implementation of project in time, budget and provided
description will satisfy [15,16,17,18]. Also these methods are naturally conservative and suitable for
low-risk projects and they are considered to have quick cost returns [2, 7, 10, 19].
Operational reasons: Benefits of IT projects are classified in three categories: tangible benefits that
are quantitative, intangible benefits that are qualitative and difficult to measure, hidden benefits that may
be considered very important or to be ignored [8]. For IT projects, proportion of advantages of tangible
benefits to intangible and hidden benefits are less. Because IT investments usually have supporting state
and their benefits are hidden. Traditional methods only measure the tangible benefits. Although the costs
are more measurable than benefits, significant amounts of capital cost of investment in IT projects are
intangible or hidden [14, 19]. Approximately 40% of the costs spent on IT projects do not be considered
in traditional budget estimates. 65% of IT managers are not able to calculate the full cost of the project
through traditional methods [19, 22]. There are many risks in IT capital investments, Such as risk
assessment, technical risk, project risk, operational risk, internal political risks, and external
environmental risks. By using traditional methods, the risks must be reflected in the discount rate [14,
20]. But only 7% of organizations consider discount rate in IT projects [9]. Even when managers are
interested to consider discount rate, the level of risk is usually calculated experimentally or based on
numbers [7]. Not considering Discount rate in projects lead to high-risk projects and highly considering
it result to conservative projects.
3. New methods of evaluating IT projects

As it is shown in Table2, many studies have acknowledged that traditional methods of evaluating IT
projects are not appropriate. Consequently lots of efforts have been done to replace these methods.
These efforts take two different approaches. The first approach is based on traditional methods and
decides to improve and resolve the major criticisms of traditional methods. The second approach does
not accept the traditional methods and use new methods for evaluating IT projects. Many new methods
for evaluating IT projects are still in the conceptual phase and are implemented limitedly [14]. These
methods introduced here generally and they are categorized base on the solutions they provide.
Strategic approach: the strategic approach was first introduced by Porter. He proposed strategic
dimensions to create competitive advantages clearly. Capital investment of IT projects should create
competitive advantages for organizations [23, 24]. The main tool for understanding the role of
technology in competitive advantage is Porter's value chain that makes alignment between capital
investment of IT projects and company goals. There is no explicit attention to financial analysis in this
approach [24]. This method provides the guidelines for selecting different investments, but it is not
appropriate when the organization wants to choose between two incompatible projects.
Information economics: Parker and Benson developed the information economy framework and
published it in 1987 [25, 26]. Information economy benefits from the allocation of point-rating scores
process for evaluating the benefits of capital investment and strategic communications with IT.
Information economy introduces a range of procedures which must be performed in the specified order:
1) forming a committee of managers, 2) consensus on intangible results, 3) quantifying importance of
benefits and risks in relative scale, 4) Estimating the probability scale in zero or 5, 5) multiplying each
estimate in weight and probability, 6) calculating total value, selecting the best alternative which take the
largest sum [7, 26]. Drawback of this method is that it needs public consensus. Other various methods
have been proposed in the context of Information economy, one of which is Oracle CB 90 that is a
software model for IT managers that have developed by Oracle to do step by step process to make
decision of capital investment in IT projects. This framework leads to a decision matrix.
Options model: options model not only are used for improving traditional methods but some new
evaluation method have been developed based on theses models. Origin of such evaluating methods is in
the evaluating of complex financial transactions such as buying and selling stocks and currency
- 11 -
transactions. These models not only pay attention to financial tracking, storage and analysis of processes
but also analyses what occurs in the case of using other options. This model changes dynamically its
structure and coefficients to improve the chances in the decision [27]. Equation of Cox-Rubenstein
model is an example of options model that is used as a base for creating new Feasibility assessment
method [7, 27]. Of course there are some doubts on this method that can be referred as follow: 1 quality
of outputs not only depend on the model but depends on the inputs. Low quality of data and conflicts
will affect outputs, 2 - Suggestions for capital investment in IT projects happens less in comparison with
the number of financial transactions, 3 - Most of these methods are complex and difficult to understand
and utilize. Before recruitment of these methods, suggestions must be approved for the applicability of
these methods in the management of portfolio investment [7]. Multilayered evaluation process: This
method uses assessment methods that more or less are hierarchical. Usually a combination of traditional
methods and strategic approach is used. it is believed that the evaluation of capital investment in IT
Projects will be effective only when the evaluation process is included in the higher levels of business
processes [21, 28] and so traditional methods are not completely put away because they can create many
benefits [6, 28]. This method usually leads to a process of two or multilayered process. In the first stage,
all capital investments that are not related to business goals of organization are eliminated. In the second
stage, selection on the remaining projects is done based on calculations obtained form modified
traditional methods. For example Earl [6] has broken the second stage to three sub stages: 1- use NPV
on the basis of tangible costs and benefits (quantitative), 2- list the intangible costs and benefits
(qualitative), 3 - analyze risks and uncertainties. Finally managers must make their decisions based on
the information obtained.
IT-BSC approach: IT projects include the production of IT application programs, programs under
Internet, work with the data bases and searching the data and so on. As mentioned in previous sections,
these projects have specific characteristics that the most important of them are intangibility and inability
to calculate the full cost of the project. Therefore, using traditional criteria that evaluate solely based on
financial criteria, is insufficient. So BSC for IT projects is specialized as IT-BSC. Use of IT-BSC from
its early concept began since 1985 [36]. Gold in 1992 and Vilkoks in 1995 made a conceptual analysis
of it that is more developed by Grembergan in 1997 and later by van Bergen and Tymerman in 1998
[37]. Also in 2005, designing performance management system aimed at monitoring and promoting the
use of IT through IT-BSC was investigated [38]. Other similar research with the aim of using the BSC in
the evaluation of organizational system performance has been done [39]. In that study all aspects that the
researchers of IT has added to general BSC model to evaluate the performance, have been investigated
and the framework for IT-BSC is presented based on measurements and strategies used in them.
Researches have shown that aspects of BSC should be changed as follows in the field of IT [39, 40, 41].
1- Company contribution: means that how much value the capital investment in IT projects creates to
business and organization.
2 - Users and customers: the evaluation of stakeholders from IT facilities which are created in the project.
3- Operational excellence: the promotion of IT products and services which are created by the assessed
project, how much depend on processes that develop and deliver software applications and information
systems.
4 - Future orientation: how much opportunities from the perspective of human resources and technology
have been created to respond to future needs of organization in the evaluated project.
Each of these aspects must be translated to indicators and measurable criteria related to the leading
project to evaluate the current state. This evaluation should be periodically repeated and objectives
already determined by using information of the best models, should be compared [37]. Other
mechanisms of communication between IT and business is development of waterfall BSC [42] In a
study in 1999 by Van der. Zee, designing "operational IT-BSC" and "developmental IT-BSC for the
development of waterfall IT-BSC has been proposed. The following figure shows the model [43].
- 12 -
Figure 1. development of waterfall IT-BSC

This model will align business strategies with IT strategies. This model show how IT creates value
through business [44]. Several studies have used BSC to evaluate projects in the field of e-commerce
[34], R & D projects [33], IT projects [29] and ICT projects [35].
COBIT Framework: COBIT framework is designed as a tool for managers in the field of IT
governance to understand and consider risk management and benefits from IT and related technologies.
This model is independent from technical IT infrastructure that is used in each organization. Its an open
standard for IT control that has developed by the "Institute of IT governance". This framework provides
substantial approach for implementing leading projects related to IT in a controlled environment [42].
COBIT framework was presented first by ISACA and ITGI in April 1996. Its goal is providing best IT
functions for IT staff, auditors and users. Then other versions are presented by development of
managerial guidelines. In May 2007, Version 4.1 has been introduced that in relation to previous
versions have more simple definition of goals and includes waterfall development of processes and
communications between business, IT goals, and IT processes [43]. Many organizations have used
COBIT to manage or control IT. This framework is a set of 138 control objectives, a set of guidelines for
auditors and collection of implementing tools [42].
IT Sources in the COBIT framework include:
Application programs; include automated systems and procedures for information processing
Information; contains organizations data such as inputs, processing and output of mechanized
systems that is used by business
Infrastructure; including technology and equipment (such as hardware, operating system, network,
multimedia and supporting environment) used for the empowerment of processing in application
programs
People; include staff required for planning, organizing, acquisition, implementation, presentation,
support, monitoring and assessment systems and internal, outsourcing or contract information services.
Department of IT in organizations should have clear vision of their IT processes for achieving their
objectives. These processes carried out business application programs by using people skills and
technology infrastructure, in order to strengthen the organizations information. These resources along
with IT processes constitute, IT Architecture of organization.
COBIT conceptual framework includes three aspects of IT processes, IT criteria, and IT
resources. Among those in the IT processes aspect, 34 processes in the management of IT in four main
areas, including planning and organizing, Acquisition & Implementation, delivery and support
and monitoring and evaluation has been determined. Figure 2 shows the relation of these four areas. In
the second aspect or aspects of information criteria, security, quality and control needs are responded by
proposing seven criteria that can be defined as what the organization wants of IT in general. These
criteria include: effectiveness, efficiency, confidentiality, integrity, availability, compliance and
reliability. In the third aspect or aspects of IT resources, five different types of sources for IT including
data, software systems, technology, facilities and human resources has been defined [44].
4. Research Methodology
To extract goals and measures in each aspect of proposed framework, goals and metrics in the first
level of COBIT IT processes has been used and they are allocated to different aspects of the framework.
- 13 -
For allocating goals to each aspect, a complete list of goals in the COBIT IT processes (including 27
goals) were given to IT experts and managers to be selected and prioritized separately for each aspect.
As in proposed framework, there are 4 aspects, SO 4 questionnaires were designed and experts opinions
were analyzed in each one. In the designed questionnaires, to determine importance of each objective in
each aspect, different questions were asked which could be answered in likert scale with 5 point:
strongly disagree (1), disagree (3), neither agree nor disagree (5), agree (7) and strongly agree (9). The
method of model evaluation was based on survey-description research method and by appropriate
statistical tests, questionnaires were analyzed.
One of the most popular ways of identification of the data statistical distribution is one-sample
Kolmogorov-Smirnov test. The Kolmogorov-Smirnov test compares the observed cumulative
distribution function for a variable with a specified theoretical distribution, which may be normal,
uniform, Poisson, or exponential [31]. According to our test results, the p-value of all questions was less
than 0.05, which showed that the distribution of them was not normal. So, a statistical non-parametric
test should be used. In this way, the binomial test is used as a non-parametric test in this paper to
determine the importance of the objectives from the view of experts. Binomial test for each question is
conducted. In this stage 7 first priority objectives in each aspect have been selected as the main goals in
each aspect because when organization can plan achieve its goal that have limited and distinguished
objectives (5 to 7) to be able to extract the key success factors among its goals [45][49]. Then by using
Friedman test, objectives were prioritized. To evaluate reliability of the questionnaires, Cronbach's Alfa
method was used.
5. Presenting integrated IT-BSC and COBIT framework to evaluate IT projects

To provide integrated IT-BSC and COBIT framework, 4 aspects including Company contribution,
users and customers, operational excellence and Future orientation which were referred in various
studies [39, 40, 41] was considered for IT-BSC. To extract goals and measures in each aspect, goals and
metrics in the first level of COBIT IT processes has been used and allocated to different aspects of the
framework. For allocating goals to each aspect, a complete list of all goals in the COBIT IT processes
(including 27 goals) are given to IT experts and managers to be prioritized separately for each aspect.
Questionnaires distributed among 40 experts and 36 of them were received. Among respondents, 20
were between 20 and 30 years old, 13 people were between 30 and 40 and 3 people were over 40 years
old and the number of persons having PhD were 10, 6 persons were PhD students, 10 of them had M.S.
and 10 were B.S.. Questionnaire distributed between people who had expertise in related field, among
them 10 had experience of 5 years, 14 had experience of 12 years and 4 hadnt determined their
experience. By using binomial test, 7 first priority objectives in each aspect have been selected as the
main goals and others were rejected because when organization can plan to achieve its goal that have
limited and distinguished objectives (5 to 7) to be able to extract the key success factors among its goals
[45].
Here the results of aspect of customers and users questionnaire analysis are shown. The value of
Coronbachs Alfa for this questionnaire was 0.874 which is more than 0.7 which shows high reliability
of results of the questionnaire. Binomial test for each question is conducted, thus one of them is
presented in the following as an example:
H0: Respond to business requirements align with business strategy isnt important as an objective of
company contribution.
H1: Respond to business requirements align with business strategy is important as an objective of
company contribution.
The result of binomial test for the customers and users aspect is shown in Table1:
- 14 -
Table 1. Binomial test for testing hypothesis for importance of each objective of customers and users aspect
Code
in COBIT
objective
PoPO1PO2 Respond to business requirements align with

AI6AI1PO10 business strategy
DS3DS1AI7
ME1
ensure integrating of application programs
AI7AI4Po2
with business processes
DS6PO5
IT share in business profitability
ensure automating of business transactions
DS5AI7PO6
and reliability of information transfer
Ensure cause and effect relation of control
AI6AI2AI1 and functional requirements with effective
automated solutions
Clarify the effects business risks in IT
PO9PO6
objectives and resources
DS1PO6PO5 Clarify expenses, benefits, strategies, policies
DS6DS2 and level of IT services
ME4ME1
Po10Po4PO1 Respond to employers governance
ME4ME1 requirements
ME1DS6PO5 Ensure continuous improvement and
ME4
readiness for change in future
DS11PO2
Optimize using of information
PO7Po4Po2 Create agility in IT processes
AI3
DS3AI3PO3 Optimize IT infrastructures, resources and
DS9DS7
capabilities
AI3AI2PO3 Create and maintain standard and integrated
AI5
application systems
AI7AI4PO6 Ensure appropriate use and performance of
DS8DS7
technology and applications solutions
DS5PO6
Ensure confidentiality of critical information
DS12DS11
Create and maintain IT skills that are able to
AI5PO7
respond to strategies
DS1AI4PO8 Ensure end users satisfaction from presented
DS8DS7DS2 level of services
DS13DS10
Ds10PO9 Protect achieved objectives
ME2
Delivery of services in required time and
PO10PO8
budget and quality standards
DS9DS5PO9 Protect IT assets
ME2DS12
AI6AI4PO8 Decrease defects of service delivery and
DS10AI7
solutions
DS4AI6PO6 Ensure minimum effect of defects and
DS12
changes of IT services in business
Create and maintain standard and integrated
AI5
IT infrastructure
Maintain information integrity and
DS5AI6
processing infrastructure
DS4AI7PO6 Ensure resistance of services and IT
DS12DS5 infrastructure from failure, delivered attack
- 15 -
Percent
of
consensus
Approved or
non approved
of H1
hypothesis
Asymp. Sig.
(p-value)
50
Isnt Approved
.142
60
Isnt Approved
.196
60
Isnt Approved
.021
70
Isnt Approved
.075
60
Isnt Approved
.095
70
Isnt Approved
.125
100
Approved
.000
30
Isnt Approved
.325
50
Isnt Approved
.025
90
Approved
.008
70
Isnt Approved
.065
70
Isnt Approved
.045
60
Isnt Approved
.125
80
Approved
.008
90
Approved
.004
80
Approved
.000
100
Approved
.000
70
Isnt Approved
.045
100
Approved
.000
80
Approved
.005
90
Approved
.002
80
Approved
.012
70
Approved
.002
80
Approved
.018
60
Isnt Approved
.621
ME2DS13
DS8DS4DS3
DS13
ME2DS11
ME4ME3
and disaster
Ensure service availability
Ensure IT compliance with rules and
contracts
100
Approved
.000
50
Isnt Approved
.725
As its mentioned above, 7 first priorities between approved objectives were selected as key objective
in the aspect. Then by using Friedman test the objectives were prioritized. Results were shown for
customers and users aspect in Table 2. Hypothesis for Friedman test are as follows:
H0: there is no significant difference between objectives
H1: there is significant difference between objectives
Table 2. Friedman test for prioritizing objectives of customers and users aspect
Objective
Score
Rank
1
2
3
4
5
6
7
Clarify expenses, benefits, strategies,

policies and level of IT services
Optimize using of information
Ensure confidentiality of critical
information
Delivery of services in required time and
budget and quality standards
Decrease defects of service delivery and
solutions
Ensure service availability
Ensure end users satisfaction from
presented level of services
13.5
15
20.7
14
19.4
8.5
26.9
Based on results of Friedman, related objectives of each aspect are determined. Further by using
metrics of first level objectives in COBIT, appropriate measures of objectives in each aspect are
determined. Results are shown in table 3.
- 16 -
Table 3. Combined framework of IT-BSC and COBIT

Customers and users
Company contribution
measures
objectives
measures
objectives
1- Respond to
C11- degree of
1- Ensure service
C21- percent of users
business
approval of business
availability
satisfaction from
requirements align
owners from IT
2- Clarify expenses,
information model and
with business
strategic and
benefits, strategies,
data availability
strategy
policies and level of IT operational programs
C22- percent of
2- ensure integrating
C12- degree of
services
satisfied IT personnel
of application
3- Delivery of services alignment with
C23- percent of
programs with
governance and
in required time and
satisfied stakeholder
business processes
business requirements
budget and quality
from quality of IT
3- ensure automating
C13- satisfaction level
standards
services, suppliers and
of business
of business from
4- Optimize using of
data integration in new
current state of projects transactions and
information
systems
reliability of
and programs
5- Decrease defects of
C24- percent of users
information transfer
C14- percent of IT
service delivery and
that are satisfied by
4- Ensure cause and
investment or
solutions
achieved performance
effect relation of
provision of
6- Ensure
C25- percent of
predetermined business control and
confidentiality of
satisfied stakeholder
functional
benefits
from service delivery in critical information
requirements with
C15- percent of IT
7- Ensure end users
agreed level
effective automated
value drivers mapped
satisfaction from
C26- percent of
solutions
to business value
presented level of
satisfied users from
5- Clarify the effects
drivers
services
service delivery in
business risks in IT
C16- percent of IT
agreed level
expenses that are stated objectives and
C27- number of
resources
by value drivers of
discontents from service
6- Clarify expenses,
business
contracts
benefits, strategies,
C17- percent of IT
C28- percent of user
policies and level of
projects that provide
satisfaction increment
business expectation in IT services
from services, systems
7- IT share in
determined timeline
and new technologies
business profitability
C29- degree of users
improvement in result
of better understanding
of systems
Future orientation
Operational excellence
measures
objectives
measures
objectives
1- Respond to
C31- percent of IT
1- Ensure continuous
C41- percent of
employers
investment or
improvement and
satisfied business
governance
owners from training of readiness for change in provision of
predetermined business requirements
future
application programs
2- Protect IT assets
benefits
2- Optimize IT
and supporting issues
3- Protect achieved
C32- percent of IT
infrastructures,
C42- number and type
objectives
value drivers mapped
resources and
of deviations from
4- Ensure
to business value
capabilities
infrastructure plan
confidentiality of
drivers
3- Ensure appropriate
C43- level of
critical information
stakeholder satisfaction use and performance of C33- percent of IT
expenses that are stated 5- Ensure resistance
technology and
from personnels IT
of services and IT
by value drivers of
applications solutions
skills
infrastructure from
4- Create and maintain business
C44- number of
failure, delivered
IT skills that are able to C34- amount of IT
supported business
attack and disaster
personnel transactions
respond to strategies
processes by old
- 17 -
infrastructure
C45- number of
systems that cover
security requirements
C46- percent of IT
compliance with rules,
regulations and
contracts
C47- degree of senior
satisfaction from
internal and external
monitoring reports
C48- percent of spend
expenses from
competitive
procurement
5- Create and maintain

standard and integrated
IT infrastructure
6- Create and maintain
standard and integrated
application systems
7- Ensure IT
compliance with rules
and contracts
C35- percent of critical

IT objectives that are
covered by risk
evaluation
C36- percent of IT risk
evaluation integrated
with IT risk evaluation
approach
C37- number of
instances were
confidential
information was
formed
C38- number of
business errors in result
of IT services errors
C39- level of
understanding from
expenses, benefits,
strategies, policies and
IT service level
C310- number of
changes in objectives
due to efficiency and
effectiveness of IT
processes
6- Create agility in IT
processes
7- Ensure minimum
effect of defects and
changes of IT
services in business
6. Evaluation of a smaple project using the proposed framework

In this section "system of electronic correspondences" in a research organization will be evaluated by
using the measures obtained in the proposed framework. This project is designed and implemented to
make the process of sending and receiving e-mails within the organization electronic and accelerates
process of responses to the e-mails. This project has established about three years and makes the
electronic communications of 450 staff and managers in Iran Telecommunication Research Center. To
evaluate the measures, a questionnaire to suit each of the measures in the various aspects was designed,
so that the degree of realization of any measure in qualitative form in mentioned project was called into
question. The questionnaires included 34 questions and were distributed between two groups of
respondents. The first group consisted of five experts including: project supervisor, two consulter,
managing director and project manager to answer three aspects of company contribution, operational
excellence and future orientation and the second group consisted of 30 users (Personnel) to respond
to questions in users and customers aspect. Responses about the agreement or disagreement were
analyzed using a five point Likert scale. The extent to which the respondents agrees or disagrees on the
realization or non-realization of the measures in the project is graded on a scale of 1 to 9, where
1=strongly isnt fulfilled, 3=isnt fulfilled, 5= almost fulfilled, 7= fulfilled, 9= strongly fulfilled. An
average, standard deviation, mod and normal weight are also calculated for each aspect. The higher the
average score, the most likely the measure is close to realization [46][47]. Further, the reliability or
internal consistency of questionnaire was assessed by Cronbachs alpha [49]. Results are shown in
Table4.
Analyses were done as follows: The mean scores from 1 to 4 means "weakness in the realization of
system measures". Mean score from 4 to 6 (except 6) is assessed as "average". The mean score equal to
6 and above means "strong" and it shows that the project fulfills the criteria to a high level. The results
of averages and weights are shown with two radar diagram in figure 2. Figure 2 shows the degree of
realization of each measure.
- 18 -
Table 4. Results achieved from the project evaluation in organization

aspect
Company
contribution
Users and
customers
Operational
excellence
Future
orientation
measure
Average
Standard
deviation
mode
Normal
weight
C11
C12
C13
C14
C15
C16
C17
C21
C22
C23
C24
C25
C26
C27
C28
C29
C31
C32
C33
C34
C35
C36
C37
C38
C39
C310
C41
C42
C43
C44
C45
C46
C47
C48
8.2
7.4
6.2
5.8
5.8
5.4
6.6
4.6
5.4
5.8
5.8
5.4
6.2
5
7
5.8
5.8
5.8
5.4
7.4
5
4.6
3.4
6.6
6.2
4.6
6.2
3.8
6.2
2.2
4.2
5.8
6.6
4.2
.98
.80
.98
.98
.98
.80
.80
1.50
.80
.98
2.04
.80
1.60
1.26
1.26
.98
.98
.98
.80
.80
1.26
1.50
.80
.80
.98
1.50
.98
.98
1.60
1.60
.98
.98
1.50
1.60
9
9
7
7
7
7
7
7
7
7
9
7
9
7
9
7
7
7
7
9
7
7
5
7
7
7
7
5
9
5
5
7
9
7
.16
.18
.13
.13
.14
.11
.15
.10
.12
.12
.12
.10
.11
.10
.10
.12
.10
.09
.10
.09
.09
.10
.07
.11
.13
.12
.15
.11
.11
.09
.14
.13
.15
.12
Overall
weight
score
6.59
5.68
5.53
5.07
As its shown in the table the company contribution aspects has higher score. This shows that the
mentioned project realize this aspect more than other ones. Realization of users and customers,
operational excellence and future orientation aspects are in the next order.
- 19 -
Figure 2. Realization degree of evaluating measures in project (scores below 4 is weak, scores between
4 and 6 is medium, and scores above 6 is strong)
7. Conclusion
Today information is the most important asset and strategic resources in any organization. Therefore,
management of IT is the major basis of organizational planning and the need to ensure the value of IT,
related risk management and information control are considered as key element in the governance of any
organization. IT project have specific characteristics such as high risk, limited return on investment,
having intangible results, high costs and hidden benefits that make the use of traditional methods to
measure the benefits of the project very difficult. This article provided a framework for evaluating IT
project with combining IT-BSC and COBIT framework. Aspects of proposed framework are chosen
from IT-BSC and objectives and related measures extracted from COBIT and validated using binomial
test and survey of experts in the field of information and communication technology. Since strategic
alignment issue is one of the most important concerns of todays IT managers, this framework is greatly
contributed to alignment of IT and business in the organization. These measures can be used to evaluate
all IT projects and assess the value of various projects to each other in an organization. For this purpose
its necessary to provide weights to four aspects of "company contribution", "users and customers",
"operational excellence" and "Future orientation" in accordance with vision and strategies of
organization and then the mean weight scores according to weight of aspects are determined. So it will
give only one score to each project that makes it comparable with other projects.
8. References
[1] Hochstrasser B., Griths C., Controlling IT investment: strategy and management, Chapman &
Hall, London, 1991.
[2] Bacon J., Why companies invest in information technology, In: Willcocks L, editor. Information
management: the evaluation of information systems investments. Chapman & Hall, London, 1994.
[3] Willcocks L., Lester S., The evaluation and management of information systems investments:
from feasibility to routine operations. In: Willcocks L, editor. Investing in information systems:
evaluation and management. Chapman & Hall, London,1996.
[4] Fitzgerald G., Evaluating information systems projects: a multidimensional approach, Journal of
Information Technology, Vol.14, pp.1730, 1998.
[5] Willcocks L., Lester S., Evaluating the feasibility of information systems investments: recent UK
evidence and new approaches, In: Willcocks L, editor. Information management: the evaluation
of information systems investments. Chapman & Hall, London, 1994.
[6] Earl JM., Management strategies for information technology, Prentice Hall, New York, 1989.
[7] Strassmann PA., The squandered computer: evaluating the business alignment of information
technologies, The Information Economics Press, New Canaan, CT, USA, 1997.
[8] Hinton CM, Kaye GR., The hidden Investments in Information Technology: the role of
organizational context and system dependency, International Journal of Information
Management, Vol.18, No.6, pp.41327, 1996.
- 20 -
[9] Ballantine J., Stray S., Financial appraisal and the ICT investment decision making process,
Journal of Information Technology, Vol.13, pp.315, 1998.
[10] Apostolopoulos TK, Pramataris KC., Information technology investment evaluation:
telecommunication infrastructure, International Journal of Information Management, Vol.19,
No.4, pp.287 96, 1997.
[11] Brealey RA, Stewart CM, Marcus AJ, Fundamentals of corporate finance, McGraw-Hill, New
York, 1995.
[12] Manigart S, Ooghe H., Investment selection methods, Management for engineers, vol.11, No.4,
pp.118, 1994.
[13] Lumby s., Investment appraisal and related decisions, Van Nostrand Reinhold, London, 1981.
[14] Clemons EK, Weber BW, Strategic information technology investments: guidelines for decision
making, Journal of Management Information Systems, Vol. 7, No.2, pp.1031, 1990.
[15] Turner JR, The handbook of project-based management. McGraw-Hill, New York, 1993.
[16] Ballantine JA, Galliers RD, Stray SJ, Information systems/technology evaluation practices:
evidence from UK organizations, Journal of Information Technology, Vol.12, No.1, pp.2941,
1996.
[17] Wateridge J., Training for IS/IT project managers: a way forward, International Journal of
Project Management, Vol.15, No.5, pp.283-288, 1997.
[18] Wright JN, Time and budget: the twin imperatives of a project sponsor, International Journal of
Project Management, Vol.15, No.3, pp.181186, 1997.
[19] Willcocks L., Information management: the evaluation of information systems investments,
Chapman & Hall, London, 1994.
[20] Dos Santos B., Justifying investments in new information technologies, Journal of Management
Information Systems, Vol.7, No.4, pp. 7190, 1991.
[21] Whiting R., Davies J., Knul M., Investment appraisal for IT systems, In: Willcocks L, editor.
Investing in information systems: evaluation and management, Chapman & Hall, London, 1996.
[22] Strassmann P., The business value of computers, The Information Economics Press, New
Canaan, CT, USA, 1990.
[23] Porter M., Technology and competitive advantage, Journal of Business Strategy, Vol. 5, No. 3,
pp.60-78, 1985.
[24] Shank JK, Govindarajan X., Strategic cost analysis of technological investments, Sloan
Management Review, Vol.34, No.1, pp.3951, 1992.
[25] Parker MM, Trainor HE, Benson RJ, Information strategy and economics: linking information
systems strategy to business performance, Prentice Hall International, 1989.
[26] Wiseman D., Information economics: a practical approach to valuing information systems, In:
Willcocks L, editor. Information management: the evaluation of information systems investments.
Chapman & Hall, London, 1994.
[27] Moad J., Time for a fresh approach to ROI, Datamation archive, Vol.41, No.
3, pp. 57-59, 1995.
[28] Willcocks L., Investing in information systems: evaluation and management, Chapman & Hall,
London, 1996.
[29] Kaplan RS, Norton DP. The balanced scorecard: measures that drive performance, Harvard
Business Review, pp.7180, 1992.
[30] Stewart W.E., Balanced scorecard for projects, Project Management Journal, Vol.32, No.1,
pp.3853, 2001.
[31] Barber, E., Miley, F., Monitoring Project Progress: More than a series of feedback loops,
Australasian Evaluation Society International Conference, October/November 2002.
[32] Stewart W.E., Balanced scorecard for projects, Project Management Journal, Vol.32, No.1,
pp.3853, 2001.
[33] Eilat H., R&D project evaluation: An integrated DEA and balanced scorecard approach, Omega,
Vol.36, No.5, pp.895-912, 2006, DOI: 10.1016/j.omega.2006.05.002.
[34] Van Grembergen, W., Saull, R., Information Technology Governance through the Balanced
Scorecard, Proceedings of the 34th Hawaii International Conference on System Sciences
(HICSS), pp 1-12, 2001.
- 21 -
[35] Milis. K, Mercken, R, The use of the balanced scorecard for evaluation of information and
communication technology projects, International Journal of Project Management, No. 22, pp.
87-97, 2004.
[36] Dickson G. Wetherbe J., The Management of Information Systems, McGraw Hill, New York,
1985.
[37] Grembergan. The Balanced Scorecard and IT Governance, Information Systems Control Journal,
Vol.2, pp.40-43 ,2000.
[38] Son, Sertac,Weitzel. Designing a Process Oriented Framework for IT Performance Management
Systems, The Electronic Journal Information Systems Evaluation, Volume 8, No. 3, pp. 219-228,
2005.
[39] Sedera; Gable;Rosemann;. A Balanced Scorecard Approach to Enterprise Systems Performance
Measurement, Proceedings of the Twelfth Australasian Conference on Information Systems.
2001.
[40] Van Grembergen, W., De Haes, S., Measuring and Improving IT Governance through the
Balanced Scorecard." Information Systems Control Journal, Vol. 2, 2005.
[41] Grembergen. IT Governance and its Mechanisms, 35th Hawaii International Conference on
System Sciences (HICCS). IEEE, 2002.
[42] Salle, Mathias, IT Service Management and IT Governance: Review, Comparative Analysis and
their Impact on Utility Computing, HP Labs Technical Report HPL-2004-98.
[43] Zee, Van der. Alignment is not enough: integrating business and IT Management with the
Balanced Scorecard, Proceedings of the 1st Conference on the IT Balanced Scorecard. antwerp,
1999.
[44] COBIT 4.1, IT governance institute, USA, 2007.
[45] Keeney, R. L., Value-Focused Thinking: A Path to Creative Decision Making .Cambridge, MA:
Harvard University Press, 1992.
[46] Ruikar, K., Anumba, C., Carrillo, P., VERDICT-an e-readiness assessment application for
construction companies, Automation in construction, Vol. 15, No.1,pp 98-110, 2006.
[47] Mohammad Kazem HAKI, Maia Wentland Forte, Service-Oriented Business-IT Alignment: a
SOA Governance Model, AISS: Advances in Information Science and Service Sciences, Vol. 2,
No. 2, pp. 51 - 60, 2010.
[48] Johannes K. Chiang, "E-Commerce Research Trend Forecasting: A Study of Bibliometric
Methodology", JDCTA: International Journal of Digital Content Technology and its Applications,
Vol. 5, No. 1, pp. 101 - 111, 2011.
[49] Zhou Ping, Le Zhong-Jian, "Analyzing Allocation Strategies of Government Information
Resources Using System Dynamic Feedback Archetypes", JDCTA: International Journal of
Digital Content Technology and its Applications, Vol. 5, No. 1, pp. 296 - 304, 2011.
- 22 -

Developing RAE1

Uploaded by

Copyright:

Available Formats

You might also like

Developing RAE1

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Developing RAE1

Uploaded by

Copyright:

Available Formats

See

Developing a Combined Framework for

Available from: Reza Samizadeh

Developing a Combined Framework for Evaluating IT Projects based on

Nasrin Dastranj Mamaghani, 2 Reza Samizadeh, 3 Fatemeh Saghafi

Keywords: IT-BSC, COBIT, Delphi Technique, IT Project, Project Management

2. Traditional methods of project evaluaiont

disadvantages too. Disadvantages of using traditional methods in evaluating IT projects can be

3. New methods of evaluating IT projects

Figure 1. development of waterfall IT-BSC

5. Presenting integrated IT-BSC and COBIT framework to evaluate IT projects

PoPO1PO2 Respond to business requirements align with

Clarify expenses, benefits, strategies,

Table 3. Combined framework of IT-BSC and COBIT

5- Create and maintain

C35- percent of critical

6. Evaluation of a smaple project using the proposed framework

Table 4. Results achieved from the project evaluation in organization

You might also like