International Journal of Project Management 20 (2002) 545–560

www.elsevier.com/locate/ijproman

R&D project efficiency management in the Spanish industry

Angel Martı́nez Sánchez*, Manuela Pérez Pérez
Area de Organización de Empresas, Centro Politécnico Superior, Marı´a de Luna, 3 Zaragoza 50015, Spain

Received 29 June 2000; received in revised form 28 November 2000; accepted 9 March 2001

Abstract
This paper shows the results from a survey to Spanish manufacturing firms on the use of indicators and techniques to analyze the
efficiency of their R&D projects. Factor analysis revealed that project’s profitability and time indicators were those which most
explained the variance of company’s efficiency analysis. On the other hand, ratios and financial analysis are the most used project
evaluation methods. The differences in the use of indicators and techniques were significative for the most R&D intensive compa-
nies. The use of some evaluation methods was related to the control mechanisms applied to R&D projects. # 2002 Elsevier Science
Ltd and IPMA. All rights reserved.
Keywords: Manufacturing; R&D; Investment; Evaluation

1. Introduction example, a study of R&D intensive USA companies [3]

R&D projects are risky investments for a private
company. For example, many studies evidence that a
40% average of new product development projects fail
to get benefits from the market [1]. As a consequence,
R&D project efficiency is a very critical issue to CEOs
and project managers. A survey carried out among the
most intensive USA companies found that the main
issue facing R&D managers was to measure and
improve project efficiency [2].
However, R&D project efficiency is not an easy mea-
surable concept. The measure of R&D efficiency
includes not only information about the output and
outcomes, such as patents, new products and profits,
but also about the process leading to them. R&D pro-
ject efficiency may be defined as the optimal use and
management of R&D inputs and processes in order to
achieve the project’s technology and market goals. An
efficiently managed project uses more productively than
the scarce resources of the R&D organization.
R&D managers are sometimes reluctanct to measure
and control the R&D work because that might hinder
people’s creativity. There are then very few empirical
studies that have analyzed the use of project efficiency
indicators and techniques in industrial companies. For
* Corresponding author. Tel.: +34-976-761-000; fax: +34-976-
761-861.
E-mail addresses: anmarzan@posta.unizar.es (A.M. Sánchez)
manuela.perez@posta.unizar.es (M. P. Pérez).
0263-7863/02/$22.00 # 2002 Elsevier Science Ltd and IPMA. All rights reserved.
PII: S0263-7863(01)00024-2
found that 60% did not measure R&D efficiency and
that only 7% regularly used some project efficiency
indicators. Other studies [4,5] also found very little use
of project efficiency analysis among industrial compa-
nies. Project managers were most sceptical about the
usefulness of efficiency analysis [6].
These empirical studies indicate that very little is
known about what managers do to analyze R&D pro-
ject efficiency. In order to give empirical evidence and to
help R&D project managers to improve their efficiency
analysis, this paper studies the use of R&D project effi-
ciency indicators and techniques in Spanish industry.
The paper is structured in the following way. Section 2
briefs the metrics available to project efficiency analysis
followed by a model of efficiency indicators from a
review of the literature. Then the paper’s research
hypotheses are established, followed by the methodol-
ogy and the discussion of the results from a survey to
Spanish industrial companies on their use of project
efficiency indicators and techniques. Finally the paper’s
conclusion is highlighted.
2. R&D project efficiency metrics
In order to consistently achieve the results of quality
products and an efficient and effective organization, the
R&D process must possess a set of necessary character-
istics: systematic, objective, accurate, precise, unambig-
546 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
uous, timely and simple [7]. Ideally, developing a R&D
process with the above characteristics both enables
informed cost–performance–schedule tradeoff decisions
at critical points and provides a basis for making
informed tradeoff decisions and identify early both
those ongoing projects with the greatest probabilities of
success and those that are destined for failure. It is
therefore argued that project evaluation must not be
seen as an end in itself, but should form part of a pro-
cess which includes the encouragement of ideas, coordi-
nation between R&D and the company and the
identification of the ‘‘needs’’ of the organization defined
in a wide sense. The need to evaluate any R&D project
in any industry is sometimes difficult, because of the
lack of objective measures, but experience shows that in
case of doubt, expenditure will cease in this area because
so often, effectiveness cannot be proved. Thus the need
to measure for value, to explore and develop objective
measures within the company, whatever the difficulty.
Whatever a R&D project is evaluated — selection,
efficiency, cancellation — managers must use metrics to
measure the project’s performance. Efficiency metrics
are usually classified into two groups: quantitative and
qualitative indicators [8]. R&D decisions are not exclu-
sively, or even mainly, adopted on the basis of quanti-
fied cash criteria. There may be many more quantified
inputs to be taken into account in evaluation and these
must feature prominently in a system for monitoring
and guiding R&D projects [9]. The quantitative mea-
sures are based on numerical variables of a R&D pro-
ject’s inputs and performance. Under this head are
collected R&D costs, capital expenditure, sales volume
and cost savings adjusted from year to year of the pro-
ject life where appropriate. The estimates of these
quantities used are the best available to the project
manager at the time of making the evaluation and will
be based on project records or opinions of experienced
technical and marketing people. Sometimes these same
people will be asked to describe in subjective probability
terms the distribution of values they expect for their
estimation of various costs, times and incomes. A com-
mon efficiency indicator is the ratio of sales or benefits
increase due to the project results divided by the invest-
ment in R&D projects [10]. The advantage of these
metrics is that they are easy to use, and are specially
recommended to later stages of the R&D project when
all quantitative information is already available. A
study of USA and German companies [11] found that
project managers emphasized the use of quantitative
ratios such as patents obtained by R&D investment.
One reason against the use of quantitative methods in
R&D is the high level of uncertainty present in research
and development. The uncertainty is, in fact, an integral
feature of the R&D work. Accordingly, methods of
R&D evaluation should accept the uncertainty and be
aimed at expressing qualitative aspects of the R&D
project that cannot be included in the quantitative indi-
cators and therefore give more information to analyze
the R&D project. The qualitative metrics are based on
subjective evaluations that may then be transformed in
numerical values using such techniques as check-lists,
scoring models or Likert scales. Some qualitative
metrics are not valued numerically but subjectively, i.e.
good, excellent, by any of the following methods:
1. Self-evaluations. Each person evaluates his or her
own contributions to the goals of the R&D pro-
ject. They are subject to biases due to self-esteem
and inaccurate self-perceptions.
2. Supervisory ratings. They are more objective than
self-evaluations, although they may also be subject
to biases. There are empirical studies that evidence
that a supervisor’s rating is correlated significantly
with the scientist’ patent productivity [12].
3. Peer-rating systems. All members of the R&D
project evaluate themselves and each other. An
assessor then correlates the data and condenses it
into an overall evaluation statement [13]. This
technique tends to undervalue any R&D effort
that it will not be measured by the assessors, either
scientific publications or patents [14].
4. External audits. They are generally more compre-
hensive. An audit is usually on the entire depart-
ment or R&D project. They are used both in
public research and in the private sector [15].
Qualitative measures are more suitable to early stages
of a R&D project when sometimes the lack of informa-
tion makes it more difficult to use quantitative metrics.
The advantage of qualitative techniques is that they
facilitate the communication and teamwork during the
project evaluation. Nevertheless, they require greater
training and expertise by project managers. Both quan-
titative and qualitative metrics are needed to analyze
R&D project efficiency [16]. Section 3 describes a model
of efficiency indicators based on a review of the litera-
ture. Both quantitative and qualitative indicators are
classified in four groups: inputs, process, outputs and
results indicators (Table 1). These groups include the
whole resources and innovation process from the early
generation idea stage to the final market evaluation of
innovation diffusion [17].
3. Model of R&D efficiency indicators
3.1. Input indicators
3.1.1. Cost and time development
To the sponsor of industrial research, time and cost
are two important elements. Time has value since it
reflects the span over which uncertainty will prevail as
to the commercial outcome. Other things being equal, a
 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560 547

project that can be completed within a specified time is 3.2. Process indicators
more valuable than one that takes longer. On the other
hand, the finantial obligation assumed in pursuing a 3.2.1. Matching to company’s strategy
research project represents the magnitude of the risk The commercial application of a successful research
involved. If the research manager has an estimate of the project has strategic implications. If a market has to be
resources required for pursuing a given line of investi- developed, the project might be less valuable than one
gation and the time required for completion, he/she can for which a known need exists, even if the latter simply
estimate the total research cost. If all other factors were maintains the company’s position in a given market. A
equal, he/she would be inclined to select the less costly project that will result in replacing an existing and
projects because it would increase the productivity ratio completely satisfactory product may not contribute as
of the project R&D investment. much as one that diversifies the company’s line. Projects
Cost and time deviations. All project evaluation that contribute most to the company’s position and
models require estimates of project cost and time, but stability with the greatest efficiency would be favoured
these estimates have proven almost as inaccurate for in similar situations. Therefore, planning for research
civilian as for military planning, especially when the and development must be consistent with the overall
projects attempt quite large technical advances. Even strategic planning of the company and must consider
when commercial projects attempt minor advances, the uncertainties in technology, distribution, customer
estimates are often still considerably wide of the mark. usage and competitor’s capabilities [7]. The practice for
All the evidence available so far demonstrates that, on matching projects against objectives of the firm and for
average, actual R&D project costs and times exceed assessing the sinergy is strongly influenced by the writ-
estimated costs and times considerably [18]. Therefore, ings of Ansoff [19] and other known strategists. R&D
both time and cost deviations have been included as must serve, in its own way, company objectives and
model indicators. Higher deviations would mean a less strategy. A better matching to company’s strategy
project efficiency. would improve the project efficiency because project
changes wouldn’t be necessary to accomodate the
adjustment to project goals.
Table 1
Model of R&D project efficiency indicators 3.2.2. Achievement of technological goals
Group Indicator Variationa The importance of having clear objectives and scope
in R&D project management is sometimes overlooked.
Input Project development cost # Definition and agreement of objectives must include a
Project total and partial development #
times
common understanding by all people involved. The
Project development cost deviations # project will become goal and results-oriented, rather
Project development time deviations # than activity-based. Having a few key objectives focuses
Process Matching to company’s strategy " the team on the target and creates commitment and
Achievement of technological goals " agreement about the project goals. The result is that the
Degree of communication between " progress of a project can be monitored effectively. Ulti-
departments mately, its success will be measured more easily
Coordination between R&D and other "
because the objectives are clearly stated at the outset
departments
Interaction between R&D and customers " of the project [20]. A faster and better achievement
and suppliers of technological goals improves the project efficiency
Top management support " because fewer changes are needed to put the project
Use of concurrent engineering " back in track.
Output Patents granted "
Scientific publications " 3.2.3. Top management support
Number of new products introduced into " Most successful R&D projects require active support
markets
from many segments of the organization. A project has
Results Cost savings due to the project results " a much better environment if it has attracted the interest
Sales increase due to project results "
and support of all related segments of the organization.
Project return on investment "
Usefulness of technologies developed " If it hasn’t, its most important first task may be to
Reaction time to competitors " attract the necessary interest and support. Early and
Level of customer satisfaction " late studies of product innovation and new technology
implementation show that top management support is a
Source: See Review of the literature in Section 3.
a
"The indicator’lue should increase to improve R&D project main project success factor which helps to reduce cost
efficiency; #The indicators value should decrease to improve R&D and time deviations and to improve project’s matching
project efficiency. to company’s strategy [21,22].
548 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
3.2.4. Degree of communication between departments
The importance of communication in organizations,
in particular its influence on the acceptance of anything
new, is well documented. Lack of timing and focused
communication has been cited as the biggest reason for
the failure of many change projects to meet their
expectations [23]. There are many reasons why commu-
nication is necessary for the successful management of a
change project. These range from ensuring an increased
understanding to eliminating waste and motivating
those involved in the change. Therefore, better commu-
nication means a more productive use of R&D and
other department’s resources.
3.2.5. Coordination between R&D and other
departments
As the interest in R&D organizations has grown, so
has the concern about providing and maintaining a
productive organizational climate for individual scien-
tists and engineers and helping R&D organizations
remain young, energetic, and even adventurous in spirit
as the organizations mature and grow older. Indeed,
the concerns about the possibility of getting into an
organizational routine are so great that it has even
occasionally suggested that all R&D organizations be
disestablished every few years and replaced with fresh
organizations that would be more willing to consider
new attitudes toward the more difficult technical pro-
blems. Besides reengineering, cooperation between
R&D and other departments is the usual way to inte-
grate R&D in the company’s innovation and technology
flow. Cross-functional teams have a positive influence
on the company’s innovation results [24]. In those com-
panies whose R&D staff members do not work closely
or responsively with marketing staff, the integration of
R&D activity with market realities is haphazard, bela-
ted, or both. Numerous case studies of successful and
unsuccessful innovation [21] come to the same conclu-
sion: the closer the link between marketing and R&D,
the greater the probability of commercialization (given
technical completion).
3.2.6. Concurrent engineering
The decision-making process in a R&D project is one
of sequential choice under conditions of substantial
uncertainty. The decision maker, starting from condi-
tions of broad uncertainty makes a stream of choices to:
gain increasingly better information about appropriate
outcomes; avoid excessive risks of unfortunate choices
along the way; and achieve progress toward a timely
completion. The progress that is made over the course
of a project in reducing uncertainty and sharpening and
improving objectives is a form of learning. The rate with
which this learning takes place is an important char-
acteristic of a R&D project that influences the way the
project can be managed. A strategy which requires a
rapid rate of learning is one of concurrent engineering
which means the simultaneous pursuit of two or more
objectives. By doing so project managers may buy
information for a better choice, hedge against the risks
of outright failure and perhaps build a broader base of
technological experience for the organization. Most of
the learning occurs in later stages which it makes a more
expensive approach [25]. R&D managers displayed a
high proclivity for pursuing parallel aproaches in
acquiring information, but a low penchant for parallel
approaches in deciding either a priori (with no experi-
mentation) or a posteriori (after experimentation)[26].
3.2.7. Interaction time between R&D organizations
Finally, the nature of technological innovation will
affect the characteristics of interorganizational linkages.
For example, products based on technological break-
throughs require substantial in-house R&D, but typi-
cally do not demand strong links with suppliers or
customers. In the automobile industry, strategic alli-
ances between competitors are increasingly common.
Often development work is divided up on the basis of
technological specialization, but the motivation is to
share cost and risk, rather than seek complementary
expertise [27]. Other scholars [28] found a positive rela-
tionship between R&D and networking. In their study
of USA technology-based firms, they found that invol-
vement of suppliers and participation in joint-venture/
strategic alliances in the R&D process was greater in
high-R&D effective organizations than in low R&D-
effective. Similarly, firms in clusters were found to be
more product innovative [29]. In such an environment
there are increased opportunities to possible interac-
tions among R&D projects such as cost or resource
utilization interaction, outcome or technical interaction,
and benefit or effect interaction [30].
3.3. Output indicators
3.3.1. Patents
A patent indicator has also been included within this
group. This is one of the few direct reflections of the
output of R&D projects, [31] although patents are a
flawed measure of innovative activity. The major pro-
blems with patents are that not all inventions are
patented and that not all patented inventions will
become innovations and successful products. In addi-
tion, patents differ in their economic impact. The quan-
tity and quality of patenting may depend on chance,
how readily a technology lends itself to patent protec-
tion, and business decision-makers’ varying perceptions
of how much advantage they will derive from patent
rights [32]. But in spite of the previous considerations,
many authors have used patent data as an output indi-
cator of the technological activities that are converted
after in products and process innovations [33]. Therefore,
 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
we have included a patent indicator as a measure of
R&D output although we admit that some companies
might not patent their findings at all. Instead of patents,
other studies of innovative productivity use the number
of new products developed as indicator.
3.3.2. Scientific publications
One of the most fundamental norms of science is for
scientists to promulgate their findings among their
peers. This can be done in a number of ways: through
reports, books, conferences, working papers, proceed-
ings, and the journal literature. Because this norm is
widely adhered to, scientists who do not disseminate
research findings via the scientific literature are widely
viewed as not really ‘‘doing science’’ [34]. However, the
use of literature indicators is mostly applicable to the
evaluation of basic research and it requires very careful
data collection. Nevertheless, including patents and
publications all together we take into account the
external published outputs of technological projects
[14].
3.4. Result indicators
3.4.1. Usefulness of technologies
The most basic property of a new product, process or
service that may result from a R&D project is the utility
of the product or technology to those who will buy and
use it. If a class of customers cannot be found who will
find the technology or product useful to them, the
resources invested in that project will be absolutely
inefficient. Utility should be judged in relation to other
technologies the customer may use to fulfill a need.
Relatively poorer utility always greatly limits the
potential of a project. A project leading to a product or
technology with no real utility is almost inevitably a
looser and other resources would have been inefficiently
invested.
3.4.2. Return on investment
Any new opportunity under consideration should
have sufficient sales/profit potential to make a sig-
nificant impact on the targeted profit center or division,
if it proves successful. It also must meet minimum cor-
porate return on investment guidelines and time frame
for producing positive cash flow [35]. A reliability study
of the potential profitability of a project is usually very
broad, going literally from present and prospective raw
materials cost and availability to present and pro-
spective future selling prices and market size.
3.4.3. Sales increase
The market gain that will result for the company if a
successful product materializes from the proposed
research project is an indicator of the return that may be
expected from a given research endeavor. Again, the
549
research manager must seek expert advice from mar-
keting personnel to determine the size of the potential
market and the share that can be expected as a result of
the selling effort, as well as the possible offsetting effect
of sales losses through product replacement. Under
similar circunstances, projects resulting in the maximum
net gain would give a more efficient use of the R&D
resources invested.
3.4.4. Reaction time to competitors
R&D projects are often greatly affected by what is or
may be happening outside the organization. Publica-
tions, new patents, new inventions, new products on the
market and other’s evident successes and failures all
have an impact on the environment in which a project is
being conducted. When a R&D project is being carried
out there is always the possibility that someone out
there is trying to do the same thing. If there is, the
question of who succeeds comes down to the question
of who will do the best job.
3.5. Methods and models of evaluation
Many R&D managers are concerned that their pro-
ject evaluation procedures are somehow ‘‘not as good as
they should be’’. Out of this concern, literally hundreds
of formulae and mathematical models have arisen for
optimizing project evaluation decisions. Some of these
formulae and models have been successfully used,
although it appears that many have not [36]. All evi-
dence points to the fact that where they are successful, it
is because the models and formulae have somehow
improved the organizational decision making behaviour.
The simplest method of formal R&D project evalua-
tion involves the completion of a checklist for each
project considered. Indicators are listed which are
believed to be important factors in determining the effi-
ciency of the R&D effort. Each project is then rated on
the basis of each indicator listed. Checklists possess the
virtues of simplicity and ease of use while providing a
formal and group structure to the process of R&D
decision making. However, checklists for different pro-
jects are difficult to compare because individual criteria
are not weighted. Scoring models attempt to remedy
this problem by assigning weights to individual criteria
and summarizing the results as a single project score
[37]. To overcome the dimensionless results of checklists
and scoring models, R&D decision makers often turn to
benefit cost ratios which have the distinct advantage of
requiring decision makers to clearly quantify their eva-
luation of a project [38]. Finally, the most quantitative
tool of R&D analysis is mathematical programming
which optimizes the expected benefits to be realizaed
from a portfolio of R&D projects while recognizing
limits on the available resources to carry out the pro-
jects [39].
550 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
4. Use of R&D efficiency indicators and models-
Research hypotheses
Section 2 has reviewed the literature to develop a list
of R&D efficiency indicators together with models to
analyze this information. This section establishes the
paper’s research hypotheses which will be tested within
a sample of Spanish manufacturing companies.
4.1. Firm size
The relationship between firm size and invention/
innovation has been a matter of long-standing debate
[40,41]. Some scholars argue that large size favours
invention because larger firms have a greater capacity to
raise capital, manage information, maintain large R&D
facilities and attract the best technical specialists. Other
scholars [42] stress the importance of smaller roles,
especially of ‘‘hi-tech’’ firms, in the process of technolo-
gical change due to their greater flexibility to adapt to
changes in external environments [43,44]. However,
large R&D departments usually have both the necessary
manpower and expertise to investigate new techniques
and the initiative to justify small savings on a cost/ben-
efit basis. As a consequence, they tend to be more
familiar with the available efficiency evaluation techni-
ques. Therefore, we would establish the two following
hypotheses:
H1a — Larger companies will use R&D efficiency
indicators more intensively than smaller companies.
H1b — Larger companies will use R&D evaluation
models more intensively than smaller companies.
4.2. R&D effort
Company’s R&D effort plays also an important role
in the R&D evaluation process. Some authors [45] talk
of an Option Theory which provides small investment in
R&D to start, and if it is successful, then the option to
invest more is taken up. Larger budgets of research and
development will require greater control to justify the
resources invested in R&D and their results. As a con-
sequence we expect that:
H2a — R&D intensive companies will have a greater
use of R&D efficiency indicators than less R&D
intensive companies.
H2b — R&D intensive companies will have a greater
use of R&D efficiency models than less R&D inten-
sive companies.
4.3. Company’s age
The number of R&D evaluation techniques is related
to the number of years spent by the R&D manager in
his/her current position [46]. Thus, with increasing
management experience, the typical R&D manager
tends to rely more on interpersonal relationships and
the knowledge of his staff’s capabilities than on for-
malized control techniques. Therefore, older companies
will have more R&D experience accumulated than
younger companies and could rely less on formal effi-
ciency analysis models for decision-making. Then, we
establish as our third hypothesis:
H3a — Older companies will have a lesser use of
R&D efficiency indicators than younger companies.
H3b — Older companies will have a lesser use of
R&D efficiency models than younger companies.
4.4. Type of R&D
Each corporation or organization has its own unique
research environment. However, three major types of
activities can usually be identified within all R&D
groups. These can be categorized as basic research,
development research and product support projects.
Basic research is not an identifiable stage in the devel-
opment of any particular consumer-oriented product
but the generation of ideas and knowledge based on the
internal basic research work or the extension of pub-
lished studies by external institutions (academia, gov-
ernment labs, etc). The type of projects that fall into the
development research category are those which result in
several well-defined end-products and other corporate
benefits, some of which may not be as well-defined; the
majority of these projects have estimates of time and
resource requirements with subjective probabilities
associated with each possible route through the life
project that result in a desired product. Finally, the
continuous introduction of competitive products into
the existing product marketplace, usually implies
substantial need for research support in order to main-
tain market shares and the implementation of new
technology.
The nature of these evaluations is different for a
research as opposed to a development project. As a
project moves from the laboratory toward the market, it
receives more intensive scrutiny from both a technical
and an economic angle. In the early research phase, the
screening of projects will probably be quick and infor-
mal, since costs at this stage are still low and predicting
outcomes is very difficult. But as projects enter the
development phase, where costs and predictability are
higher, they require a far more detailed process of eco-
nomic evaluation. Therefore we proposed the following
hypothesis:
H4 — The use of some R&D efficiency indicators will
vary among companies with their portfolio of R&D
projects.
 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
5. Empirical study in the Spanish industry. Method-
ology
A mail survey was carried out in the first quarter of
the year 2000 to the 423 Spanish industrial companies
that received public financial support to accomplish
R&D projects of over a million euros in the 1990–1999
period. Companies’ data were obtained from the Tech-
nological Development Agency (CDTI) that had
financed those projects. The questionnaire was sent to
each company’s technology or R&D manager. After a
second roundup, 87 companies returned useful ques-
tionnaires for this study.
The response rate (20.5%) is low but not very differ-
ent from other analysis of R&D project evaluation
techniques [36,46]. Nevertheless, we think that the sam-
ple is still a good representation of the population
because other 78 companies (18.4%) said they didn’t
carry out R&D projects on a regular basis and therefore
didn’t use sistematically R&D efficiency indicators.
Consequently, the representation of the sample would
rise up to 25.2%. We think that other companies in the
population didn’t have a systematic R&D efficiency
analysis. As a consequence the actual representation of
the sample could be even higher.
Besides there are not significant differences on indus-
try and size distribution between the whole population
and the surveyed sample. Thirty-nine questionnaires
were from large companies (more than 250 employees)
and 48 from small and medium sized companies. The
average technological effort (%R&D/Sales) in the sam-
ple was 5%, and 33 companies did an effort over aver-
age. The main industries involved in this study are
chemistry and pharmaceutical (22.9%), manufacturing
machinery (20.6%) and electronics and computer
(17.2%).
The purpose of the survey was to study the use of the
R&D efficiency indicators among these industrial com-
panies. The survey is included in the Appendix. The
R&D managers were asked about the use of efficiency
indicators, evaluation models, and control techniques
within their companies. Next sections describe and
discuss the survey results.
6. Results
6.1. Project efficiency analysis making
Seventy seven percent of surveyed companies did
project efficiency analysis. There are not significant dif-
ferences in the use of efficiency analysis between large
(74%) and medium sized companies (79%), and
between more R&D intensive companies (82%) and less
R&D intensive companies (74%). A logistic regression
analysis has been carried out to test the relationship
551
between the making of efficiency analysis and the com-
panys’s age, size and R&D effort. Table 2 shows the
results. The only significant variable is the R&D effort.
The relationship is positive, which means that the
probability to do efficiency analysis of technological
projects is higher in the more R&D intensive companies
than in less R&D intensive companies. The manage-
ment of R&D requires an increase in R&D resources
productivity in order to gain competitiveness. Project
efficiency analysis helps to identify waste of resources
that are slender in smaller firms.
6.2. Use of project efficiency indicators
Tables 3 and 4 show, respectively, the percentage of
surveyed companies that used the quantitative and
qualitative efficiency indicators, as well as the mean and
standard deviation of their intensity of use in a five-step
Likert scale from 1 (nothing) to 5 (always). The three
most used quantitative indicators are: the project devel-
opment cost (99%), the deviations in development costs
(98%), and the project development time (97%). The
two less used quantitative indicators are the number of
scientific publications (48%) and the number of patents
obtained (71%). Both probability and intensity of use of
each quantitative indicator are strongly correlated. The
two most intensively used indicators are the project
development cost (4.32 in a five-step Likert scale) and
the project development time (4.18).
With respect to the qualitative indicators, Table 4
shows that the three most used indicators are: the use-
fulness of the developed technologies (98%), the degree
of achievement of the project technological objectives
(97%), and the project focus on the company’s strategy.
The two less used indicators are the reaction time to
competitors (83%) and the interaction time of R&D
employees with other departments (80%). Similarly to
the quantitative indicators, there is a positive correla-
tion between the use and the intensity of use of the
qualitative indicators in the efficiency analysis of the
technological projects.
Using the comparison means t-test, it has contrasted
the significant differences in the probability and inten-
sity of use of the project efficiency qualitative and
Table 2
Logistic regression on the efficiency analysis making in the surveyed
companies
Constant Size R&D Age
effort
Coefficient 0.277 0.002 0.152* 0.013
Wald value 0.251 0.721 3.501 1.202
2 Log Likelihood=86. 904 Chi-square=6.906* n=87
Source: Own elaboration.
*P <0.1.
552 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560

quantitative indicators, according to company’s size
(large or small and medium sized firm), R&D effort
(%R&D/Sales over and below 1%) and age (number of
years over and below the 33 years average). There were
significant differences in the percentage of companies
that use these indicators according to the company’s
R&D effort but not to their size or age. Table 5 shows
the indicators whose differences of use have been found
significant according to R&D effort. The most sig-
nificant difference is for the number of patents indi-
cator: companies with higher R&D effort use this
indicator in much greater proportion than less R&D
intensive companies.
On the other hand, Table 6 shows the significative
differences in the importance of indicators to control
project efficiency according to company’s R&D effort.
Patents and scientific publications are indicators more
important to R&D intensive companies, while cost sav-
ing and strategy focus are important to less R&D
intensive companies. Regarding company’s size and age
there are also some significative differences in the
importance of project efficiency indicators. Generally,
large companies assigned more value to the quantitative
than to the qualitative indicators, while small and med-
ium sized companies did it otherwise. However, the only
project efficiency indicator which had a significative dif-
ference at the 90% level between large and small and
medium sized companies (2.90 as compared with 2.35,
respectively) is the number of patents. With respect to
the company’s age, younger companies ( < 10 years in
business) valued more the indicators of project cost and
project cost deviation than older companies (4.52 and
4.17, respectively, as compared with 4.08 and 3.64), with
significant differences at the 95% level. There are also
significant differences at the 90% level on the project
indicator of concurrent engineering between younger
and older companies (3.19 as compared with 2.72). Both
results indicate that younger companies may be more
cost-focused than larger companies in order to compete
in the marketplace, and then they give more importance

Table 3
Use of R&D project efficiency quantitative indicators

Percentage of Average Standard

user companies of use indicator

deviation
Project cost development 99 4.32 0.97
Project cost development deviations 98 3.93 1.17
Project total development time 97 4.18 1.05
Cost savings due to the project results 95 3.72 1.25
Sales increase due to the project results 93 3.78 1.24
Project time development deviations 92 3.80 1.31
Project stages development time 90 3.55 1.34
Project return on investment 90 3.55 1.34
Number of new products introduced into markets 89 3.55 1.29
Use of concurrent engineering in the project 86 2.98 1.26
Number of patents granted 71 2.60 1.34
Scientific publications 48 1.86 1.12

The intensity of use has been measured with a five-step Likert scale from 1 (nothing) to 5 (always). The indicators are enumerated in decreasing
order of using companies. Source: Own elaboration.

Table 4
Use of R&D project efficiency qualitative indicators

Indicator Percentage of Average Standard

user companies of use deviation

Usefulness of technologies developed 98 3.92 1.03

Degree of achievement of technological goals 97 3.97 1.02
Project matching to company’s strategy 94 4.02 1.16
Coordination between R&D and other departments 94 3.74 1.19
Degree of communication between departments during the project 93 3.36 1.13
Level of customer satisfaction with the project results 92 2.93 1.10
Top management support to the R&D project team 91 3.61 1.30
Reaction time to competitors 83 2.98 1.26
Amount of interation time between customers, suppliers and R&D employees 80 2.64 1.19

The intensity of use has been measured with a five-step Likert scale from 1 (nothing) to 5 (always). The indicators are enumerated in decreasing
order of using companies. Source: Own elaboration.
 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560 553

to project costs and to project flexibility techniques such
as concurrent engineering that allow to reduce project
costs.
After analyzing the use and importance of each pro-
ject indicator in the surveyed companies, a contingency
analysis was carried out to test some relationship
hypotheses on the use of indicators. Table 7 shows the
results. All the Chi-square and Likelihood Ratio values
are significative which validates the dependence rela-
tionships proposed between indicators. The strongest
dependence relationship is between project cost and cost
deviation (Cramer’s V over 0.5).
Then a logistic regression carried out to analyze the
probability of use of each project efficiency indicator
according to company’s age, size, and R&D effort. The
results indicate that size is the only significative variable
to all but a few project indicators: development cost,
concurrent engineering, patents, and scientific publica-
tions. The variable size is negatively correlated which
means than the probability of use of most of the project

Table 5
Significant differences in the average use of project efficiency quantitative and qualitative indicators according to the company’s R&D effort

Indicator % of use in companies % of use in companies

with a R&D/Sales less than 1% with a R&D/Sales over 1%

Patents obtained 43 77***

Number of new products introduced into markets 71 92**
Sales increase due to the project results 79 96**
Project return on investment 71 93**
Reaction time to competitors 64 86**

The significant degree in the mean differences have been obtained with the t-student test Source: Own elaboration.
**P <0.05.
***P <0.01.

Table 6
Significative differences in the importance of project efficiency quantitative and qualitative indicators according to the company’s R&D effort

Indicator Importance in companies with Importance in companies with

R&D/Sales less than 8% R&D/Sales over or equal than 8%

Patents obtained 2.42 3.14**

Scientific publications obtained 1.69 2.36**
Cost savings due to the project results 3.94 3.09***
Project matching to company’s strategy 3.89 4.41*

The intensity of use has been measured with a five-step Likert scale from 1 (nothing) to 5 (always). The significant degree in the mean differences
have been obtained with the t-student test. Source: Own elaboration.
*P <0.1.
**P <0.05.
***P <0.01.

Table 7
Dependence relationships tested between some project efficiency indicators

Relationships Chi-square Likelihood Ratio Cramer’s V

Total time–time deviations 68.067**** 55.356**** 0.442****

Partial times–time deviations 68.329**** 58.426**** 0.443****
Cost–cost deviations 117.154**** 84.651**** 0.580****
Concurrent engineering–cost savings due to project results 29.387** 33.063*** 0.291**
Patents–scientific publications 49.048*** 51.542**** 0.375****
Sales increase–project return on investment 72.485**** 66.764**** 0.456****
Cost savings–project return on investment 47.954**** 39.937*** 0.371****
R&D coordination–communication with other departments 80.866**** 72.849**** 0.482****
Usefulness of technologies–customer satisfaction 49.301**** 36.602*** 0.376****
Time with customers–reaction time to competitors 39.073*** 36.695*** 0.335***

Source: Own elaboration.

**P <0.05.
***P <0.01.
****P <0.001.
554 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560

indicators are greater in smaller companies than in are listed in Table 9. All eigen values exceeded one, and
larger ones. the cumulative percentage of total variance explained
Finally, a linear regression analysis tried to explain was 70%. There was a high degree of convergence
the differences in the percentages of average use of pro- within each factor, the lowest factor loading within a
ject efficiency indicators, according to the company’s factor was 0.51, with most exceeding 0.65. Also, there
age, size, and R&D effort. Table 8 shows the results. was a high degree of divergence across factors as indi-
The adjustment of the model is small but significative. cated by the lack of high cross-loading of any item on
Size is the only significative variable, which confirms more than one factor.
again the negative relationship between size and use of The seven factors were named Profitability, Time,
project indicators. Small and medium sized companies Network, Cost, Interface, Diffusion, and Strategy. The
make a greater use of the project efficiency indicators first is comprised of sales increase, return on investment,
than large companies. cost savings, customer satisfaction, and new products
introduced. High usage of these variables indicates that
6.3. Factor analysis of efficiency indicators the focus of efficiency relies on finantial and market
values. The second factor is clearly centered on project
The 21 indicators were subject to principal compo- development times. The third factor, Network, is con-
nents factor analysis with varimax rotation. The seven stituted of communication between departments, coor-
factor solution was chosen for several reasons. Results dination with departments, and usefulness of project
results. These three have in common the network
mechanisms, which are necessary to carry out the pro-
Table 8 ject. The four factor is centered on project development
Linear regression of the average use of indicators
costs. The fifth factor, consisting of interaction between
Constant Size R&D effort Age R&D and customers, reaction time to competitors, and
concurrent engineering, is centered on interfaces among
Coefficient 0.880*** 0.009*** 0.001 0.007
t-student 28.909 5.456 0.596 1.246
companies or tasks in project management. The sixth
factor is comprised of patents and scientific publica-
R=0.524 R2=0.275 F=10.492 P=0.000 n=87
tions, that accounts for the project results diffused on
Source: Own elaboration. papers and patent documents. Finally, the seventh fac-
***P <0.01. tor is constituted of achievement of technological goals,

Table 9
Factor analysis of R&D efficiency indicators

Factor Eigen-value Pct. of var Indicators Loading

Profitability 6.427 27.94 Sales incease due to the project results 0.839
Project’s return on investment 0.684
Cost savings due to the project results 0.670
Level of customer satisfaction 0.606
Number of new products introduced 0.592
Time 2.232 9.70 Project development total time 0.785
Project time deviations 0.776
Project development partial times 0.775
Network 1.836 7.98 Degree of communication between departments 0.867
Coordination between R&D and other departments 0.832
Usefulness of technologies developed 0.628
Cost 1.764 7.66 Project development cost 0.823
Project development cost deviations 0.768
Top management support 0.510
Interface 1.459 6.34 Interaction between R&D and customers 0.709
Reaction time to competitors 0.691
Use of concurrent engineering 0.551
Diffusion 1.240 5.39 Scientific publications 0.763
Patents 0.757
Strategy 1.083 4.71 Achievement of technological goals 0.788
Matching to company’s trategy 0.563

Source: Own elaboration.

 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
and matching the company’s strategy which suggest an
strategic project dimension.
6.4. Project efficiency evaluation models
This section studies the use of models to organize the
information supplied by the project efficiency indicators.
First, Table 10 indicates the percentage of companies
that used each method and their importance in a five-
step Likert scale. The two most used methods are the
ratios (87% of companies and 3.43 of importance
value), and the financial analysis (83% and 3.17). The
less used method is mathematical programming (26%
and 1.47). It is important to underline that more than
half of the companies elaborate their own check-list.
This technique allows to integrate qualitative and
quantitative information, which it is very useful for
those projects environments, like R&D or new technol-
ogy implementation, where there is lack of quantitative
information or there is need to take strategic informa-
tion into account. These results are similar to earlier
surveys of R&D management techniques [36,46].
The differences in the percentage of companies that
use each technique are not significant between large,
small and medium sized companies, but there are some
significative differences with respect to company’s age
and R&D effort. For example, scoring models are used
by 61% of the companies that invest over 5% in R&D,
while it is only used by 41% of companies that invest
less than 5%, and this difference is significative at the
90% level. Concerning the age of the company, Table 11
indicates that these techniques are more used by older
companies than by younger ones.
On the other hand, the differences in the degree of use
of each evaluation method by the large and the small
and medium sized surveyed companies are not sig-
nificative. However, companies which invest over 5% in
R&D are more intensive users than those which invest
less than 5%. The only significative difference, at the
95% level, is for the use of ratios (3.79 in comparison
with 3.20). Regarding the variable age, those companies
older than average use more intensively these techniques
Table 10
Use of project efficiency evaluation methods
Technique Percentage of Average Standard
user companies of use deviation
Ratios 87 3.43 1.34
Finantial analysis 83 3.17 1.43
Check-list 66 2.62 1.50
Scoring 48 1.69 0.85
Mathematical 26 1.47 0.93
programming
Note: The intensity of use has been measured with a five-step Likert
scale from 1 (nothing) to 5 (always). The indicators are enumerated in
decreasing order of using companies. Source: Own elaboration.
555
than younger companies. The only significant differ-
ence, at the 95% level, is for the use of scoring methods.
A logistic regression was carried out to study the
probability of use of each evaluation method according
to company’s age, size and R&D effort. The results
indicate that the Ratios is the only method whose
probability of use may be explained by the proposed
variables. In this model, the variable size has a Wald
value of 3.928, significative at the 90% level. Size and
R&D effort do not explain the use of Ratios or any
other project evaluation method. Neither has a sig-
nificant relationship been found after separating the
company’s R&D effort into their components of basic,
development, and product support or new technology
implementation.
However, significant results have been found in a
contingency analysis to test the relationships between
the intensity of use of project efficiency indicators and
methods. Table 12 shows the results: the use of ratios
and financial analysis has a strong dependence relation-
ship to the project return on investment indicator; on
the other hand, the suitability of the R&D project to the
company’s strategy is related with the use of check-list
and scoring models because the development and
implementation of these project evaluation methods
require a common link among their different indicators.
6.5. Project efficiency control techniques
The last section of the empirical analysis relates to the
control mechanisms used to manage a negative evolu-
tion in the project efficiency indicators. Table 13 briefs
this section results. The two most used techniques,
besides increasing the control on the project, is to apply
more resources (94% of companies use it with a value of
3.20 out of 5), and to reassign people to the project
(91% of companies and a value of 2.79). No sig-
nificative differences have been found within the per-
centages and degree of use of these mechanisms according
to company’s age and size. However, the R&D effort
variable introduces some significative differences: at the
Table 11
Significative differences in the use of project efficiency evaluation
methods according to companys’s age
Technique % of user % of user
companies with companies with
age older than average age younger than average
Check-list 56 77**
Scoring 35 64***
Mathematical 25 28*
programming
Source: Own elaboration.
*P <0.1.
**P <0.05.
***P<0.01.
556 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560

95% level, the analysis of the anticipated cancellation of
the R&D project was more used in the companies with
an R&D effort over 5% than in the less R&D intensive
companies (Table 14).
Finally, Table 15 shows the results of the dependence
relationship test between the use of these project control
mechanisms and the project efficiency evaluation meth-
ods. The anticipated cancellation is positively related to
the use of such methods, like check-lists and scoring,
that allow inclusion of a greater number of indicators in
the decision making process. It is also positively related
to the use of mathematical programming which permits
analysis of the impact of any individual project within a
project portfolio. On the other hand, the mechanisms of
increasing control and reassigning staff seem to be more
related with the simple and individualistic evaluation
methods like ratios.
7. Discussion
Selecting the appropriate set of indicators and meth-
ods that affect the ability to maximize efficiency in R&D
project management from a widely available ‘‘menu’’ is
troublesome. We found that a broad set of 21 indicators
can be classified empirically into seven factors. These
seven factors differed in their relative impact to explain
the efficiency analysis in the surveyed companies.
Profitability was found to be a consistent predictor of
the project efficiency evolution. Profitability was a
factor consisting of project’s return on investment, sales
increase and cost savings due to the project results, level
of customer satisfaction, and the number of new pro-
ducts introduced. It was named Profitability because the
common theme connecting these five indicators is eco-
nomic profit generated by the project’ success. Specifi-
cally, the use of each of the five components of this
factor has the potential to show directly the positive or
negative change in the project’s economic returns.
The factor Time was also an important predictor of
project efficiency evaluation (eigen value over 2). Factor
analysis then suggested that companies look for effi-
ciency mainly throughout the project’s profitability and
time development. Even though cost and strategy indi-
cators were also mostly used by the surveyed compa-
nies, their influence on the efficiency analysis as less
important than profitability and time indicators. The
literature shows that companies analyze R&D project’s
proposals on such premises as economic returns, market
attractiveness or strategy networking [36,46]. However,
some of these factors become less relevant while the
project is being developed. This is important because it
shows that companies must perform a dynamic project
evaluation. Some indicators will still be useful after
starting the project but others will be either less or more
relevant. For example, project development cost is an
important R&D selection criteria but once the project is
approved, this indicator is no longer operative since the
resources have already been invested. However, project
development time or profitability are examples of R&D

Table 12
Test of dependence relationships between project evaluation methods and indicators

Relationships Chi-square Likelihood Ratio Cramer’s V

Ratios–project return on investment 37.903*** 42.183*** 0.330***

Finantial analysis–project return on investment 36.486*** 38.934*** 0.324***
Finantial analysis–project matching to company’s strategy 29.476** 27.953** 0.291**
Check-list–project matching to company’s strategy 23.954* 26.231* 0.262*
Scoring–project matching to company’s strategy 24.881** 29.804*** 0.309**

Source: Own elaboration.

*P <0.1.
**P <0.05.
***P <0.01.

Table 13
Use of project control mechanisms in presence of negative efficiency indicators

Method Percentage of Average Standard

user companies of use deviation

Supervision increase over the project 98 4.00 0.93

Increase of resources devoted to the project 94 3.20 0.95
Reassignment of people to the project 91 2.79 1.05
Modification of project goals 87 2.93 1.10
Analysis of project cancelation 84 2.95 1.24
No changes in the project development 62 2.24 1.22

The intensity of use has been measured with a five-step Likert scale from 1 (nothing) to 5 (always).Source: Own elaboration.
 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
indicators which still are informative after the project’
start. Either publications, patents or strategy factors
seemed to be less important for efficiency analysis.
Another result from this survey with implications for
industrial companies is that neither company’s age or
size explain the differences in the use of efficiency indi-
cators. A company’s R&D effort is the only explaining
variable. R&D intensive companies had a greater use of
effiency indicators. This result is supported by other
empirical studies on R&D evaluation. The larger and
more diversified R&D budget, the greater need to ana-
lyze several dimensions of R&D project efficiency. No
matter the company’s size or age but its R&D effort to
create the need to evaluate the project. A company
willing to increase its R&D budget should improve its
knowledge base on R&D project evaluation. Those
companies more intensive in R&D activities made a
more extensive use of R&D evaluation indicators and
methods.
The type of R&D carried out by the company did not
seem to have an influence on the use of R&D project
indicators and evaluation methods. This evidence didn’t
support the paper’s fourth hypothesis. A plausible
explanation for this result is the background of the
sample. These companies received financial support to
carry out R&D development projects but not basic
research or new technology implementation. The dis-
tribution of their R&D activities is focused on develop-
ment work and applied research. However, evidence
Table 14
Significative differences in the study of R&D project cancelation
according to the company’s R&D effort
R&D/Sales R&D/Sales over
less than 5% or equal than 5%
Percentage of companies 78 94**
which study cancelation
Intensity of use 2.74 3.30**
The intensity of use has been measured with a five-step Likert scale from
1 (nothing) to 5 (always). The signicant degree in the mean difference
have been obtained with the student-t test. Source: Own elaboration.
**P <0.05.
Table 15
Relationships tested between project control mechanisms and efficiency evaluation methods
Relationships
Anticipated cancelation–check-list
Anticipated cancelation–scoring
Anticipated cancelation–mathematical programming
Supervision increase–ratios
Supervision increase–financial analysis
Reassignment of people–financial analysis
Note: Level of significance Source: Own elaboration.
*P <0.1.
**P <0.05.
***P <0.01.
557
from the literature states that the use of R&D indicators
and evaluation methods is dynamic and diversified with
the company’s R&D portfolio. For example, the avail-
ability and reliability of data, the level of detail for the
statement of project goals, the time horizon for project
activity, the criteria for successful project completion
and the structure of the R&D and supporting groups,
among other factors, are quite different in ‘‘R’’ and ‘‘D’’
environments. Also, corporate and divisional R&D
groups have differing missions and project portfolios
owing to their respective roles in the organization’s
overall technology strategy.
The survey results also show that R&D efficiency
analysis was not performed exclusively in large compa-
nies but in smaller companies as well. The use of indi-
cators and evaluation methods do not have a size
barrier, since the knowledge base necessary to do this
analysis is updated and accessible in the literature.
However, the use of evaluation methods seem to be
limited to such simple models as Ratios, financial ana-
lysis or check-lists. There are several reasons why the
more complicate models have not found extensive use:
many of the mathematical and analytical models fail to
recognize that R&D is esentially a process of buying
information, that unsucessful projects can provide
valuable information, and as a result that the real task is
to facilitate sequential decision making under condi-
tions of uncertainty; the models often rest on overly
optimistic estimates that are not very reliable; and
finally the application of the more sophisticated models
is not cheap because they need a lot of information.
Nevertheless, as the manager becomes familiar with the
efficiency model analysis, the sophistication of use and
the subsequent analyses might increase. Without ques-
tion, the simple evaluation methods have been useful,
since they have forced managers to make their assump-
tions explicit.
Many managers still contend that their environment
is ‘‘unique’’ and not amenable to ‘‘generic’’ manage-
ment science models and techniques. Therefore, the
organizational context in which R&D resource alloca-
tion occurs must be considered by management scientist
Chi-square Likelihood Ratio Cramer’s V
26.399** 32.551*** 0.275**
26.640*** 25.675** 0.319***
28.475** 25.448* 0.286**
26.350** 30.173** 0.275**
25.788* 33.416*** 0.272*
29.536** 31.143** 0.291**
558 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560

in the development of appropriate methods. For exam- 8. Conclusion

ple, if a management science technique’s applicability is
limited to R&D, it is more difficult to gain its accep-
tance by the company at large. This acceptance is cri-
tical since groups external to R&D often fund and/or
approve R&D projects. Also, some groups sponsoring
or funding R&D will often impose the usage of certain
project management techniques. The majority of R&D
managers surveyed used those techniques which were
accepted and understood by other functional managers
(e.g. marketing, finance and production) they interface
with in their organizations. Notable examples are the
usage of financial measures for R&D project selection
and evaluation which were originally chosen for the
capital budgeting process. R&D managers need to be
well-versed in the capital budgeting techniques used by
their companies since these are often the basis for the
finantial evaluation of R&D projects. The initial train-
ing for R&D managers in project management should
provide an overview of the available methods, descrip-
tion of their data requirements, advantages/limitations
and organizational ‘‘fit’’ considerations. This back-
ground can provide direction for selecting more specific
training as needed.
This paper briefs and discusses the results from a
survey to Spanish industrial companies on their R&D
project efficiency analysis. Seventy seven percent of the
surveyed companies carried out efficiency analysis,
being the probability greater in the more R&D intensive
companies. Both quantitative and qualitative indicators
were similarly used. The three most important indica-
tors are related with costs and development times, and
with the usefulness and achievement of the project
technological goals. Some significative differences have
been found within the use of some of these indicators
according to the company’s R&D effort.
Regarding the project efficiency techniques, the most
used are the ratios and the financial analysis, with sig-
nificative greater differences of use in the older compa-
nies. No significative differences have been found
regarding other variables such as size, R&D effort
or type of R&D. Finally, the differences in the use
of project control mechanisms have been significant
with the company’s R&D effort, as well as the relation-
ship to the use of some project efficiency evaluation
methods.

Appendix A. Survey to companies

1. Please, indicate the distribution percentage of the company’s R&D investment among the following projects: Basic
Research, Applied Research, Development, and New Technology Implementation.
2. Are R&D project efficiency evaluations being perform systematically within your company? Yes No
3. Please, value with a five-step scale from 1 (nothing) to 5 (always) the use of the following indicators to analyze the
efficiency evolution of the company’s R&D projects.
Project cost development 1 2 3 4 5
Project total development time 1 2 3 4 5
Project cost development deviations 1 2 3 4 5
Project time development deviations 1 2 3 4 5
Matching to company’s strategy 1 2 3 4 5
Achievement of technological goals 1 2 3 4 5
Degree of communication between departments 1 2 3 4 5
Coordination between R&D and other departments 1 2 3 4 5
Interaction between R&D and customers and suppliers 1 2 3 4 5
Top management support 1 2 3 4 5
Use of concurrent engineering 1 2 3 4 5
Patents granted 1 2 3 4 5
Scientific publications 1 2 3 4 5
Number of developed ideas 1 2 3 4 5
Number of new products introduced into markets 1 2 3 4 5
Quality and excellence rewards 1 2 3 4 5
Cost savings due to the project results 1 2 3 4 5
Sales increase due to project results 1 2 3 4 5
Project return on investment 1 2 3 4 5
Usefulness of technologies developed 1 2 3 4 5
Reaction time to competitors 1 2 3 4 5
(Continued on next page)
 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560 559

4. Please, value with a five-step scale from 1 (nothing) to 5 (always) the use of the following evaluation methods to
organize efficiency information from the company’s R&D projects.
Ratios 1 2 3 4 5
Financial analysis 1 2 3 4 5
Check-list 1 2 3 4 5
Scoring 1 2 3 4 5
Mathematical programming 1 2 3 4 5

5. Please, value with a five-step scale from 1 (nothing) to 5 (always) the use of the following project control
mechanisms whenever the R&D project’s evolution is negative.
Supervision increase over the project 1 2 3 4 5
Increase of resources devoted to the project 1 2 3 4 5
Reassignment of people to the project 1 2 3 4 5
Modification of project goals 1 2 3 4 5
Analysis of project cancelation 1 2 3 4 5
No changes in the project development 1 2 3 4 5

References [17] Poole M. Organizational change and innovation processes:

[1] Stevens G, Burley J. 3.000 Raw ideas=1 commercial success!
Research and Technology Management 1997;40:16–27.
[2] Burkhart R. Survey results to the three ‘biggest’ problems.
Washington, D.C., USA: Industrial Research Institute, 1995.
[3] Schainblatt A. How companies measure the productivity of
engineers and scientists. Research Management 1982;25:10–18.
[4] Moser M. Measuring R&D performance in R&D settings.
Research Management 1985;28:31–3.
[5] Higgins J, Watts K. Some perspectives on the use of management
science techniques in R&D management. R&D Management
1986;16:291–6.
[6] Brown M, Svenson R. Measuring R&D productivity. Research
and Technology Management 1998;41:30–5.
[7] Casler J. Management of research and development projects in
small technical services companies. Project Management Journal
1997;28:19–24.
[8] Werner B, Souder W. Measuring R&D performance- state of the
art. Research and Technology Management 1997;40:34–42.
[9] Bradbury F, Gallagher W, Suckling C. Qualitative aspects of the
evaluation and control of research and development projects.
R&D Management 1973;3:49–57.
[10] McGrath M, Romeri M. The R&D effectiveness index. Journal
of Product Innovation Management 1994;11:213–20.
[11] Werner B, Souder W. Measuring R&D performance — U.S. and
German practices. Research and Technology Management
1997;40:28–32.
[12] Keller R, Holland W. The measurement of performance among
research and development professional employees, a longitudinal
analysis. IEEE Transactions on Engineering Management
1982;29:54–8.
[13] Meinhart W, Pederson J. Measuring the performance of R&D
professionals. Research and Technology Management
1989;32:19–21.
[14] Herbertz H, Müller-Hill B. Quality and efficiency of basic
research in molecular biology, a bibliometric analysis of thir-
teen excellent research institutes. Research Policy 1995;24:959–
79.
[15] Rubenstein A, Geisler E. Evaluating the outputs and impacts of
R&D/innovation. International Journal of Technology Manage-
ment 1991;4:181–204.
[16] Hauser J, Zettelmeyer F. Metrics to evaluate R,D&E. Research
and Technology Management 1997;40:32–8.
theory and methods for research. London: Oxford University
Press, 2000.
[18] Rubenstein A, Schröder H. Managerial differences in assessing
probabilities of technical success for R&D projects. Management
Science 1977;24:137–48.
[19] Ansoff H. Corporate strategy. London: Penguin, 1968.
[20] Clarke A. A practical use of key success factors to improve the
effectiveness of project management. International Journal of
Project Management 1999;17:139–45.
[21] Freeman Ch. The economics of industrial innovation. Penguin,
London, 1974.
[22] Martı́nez AUrbina, O Pérez M, Alonso I. Alta dirección y auto-
matización flexible: una nota. Revista Asturiana de Economı́a
1998;13:189–95.
[23] Diekmann M. Making new technology investment pay off.
Managing Office Technology 1996;41:11–14.
[24] Atahuene K, Evangelista F. Cross-functional influence in new
product development: an exploration study of marketing and
R&D perspectives. Management Science 2000;46:1269–84.
[25] Abernathy W. Some issues concerning the effectiveness of parallel
strategies in R&D projects. IEEE Transactions on Engineering
Management 1971;18:80–9.
[26] Moskowitz H. R&D managers’ choices of development policies
in simulated R&D environments. IEEE Transactions on Engine-
ering Management 1972;19:22–30.
[27] Alvarez M, González P. Strategic alliances, organisational learn-
ing and new product development: the cases of rover and seat.
R&D Management 1999;29:391–404.
[28] Gupta A, Wilemon D, Atuahene-Gima K. Excelling in R&D.
Research-Technology Management 2000;43:52–8.
[29] Baptista R, Swann P. Do firms in clusters innovate more?
Research Policy 1998;22:525–40.
[30] Fox G, Baker N, Bryant J. Economic models for R&D project
selection in the presence of project interactions. Management
Science 1984;30:890–902.
[31] Griliches Z. Patent statistics as economic indicators, a survey.
Journal of Economic Literature 1990;28:1661–707.
[32] Arundel A, Kabla I. What percentage of innovations are paten-
ted? empirical estimates for European firms. Research Policy
1998;27:127–41.
[33] Ernst I. The use of patent data for technological forecasting, the
diffusion of CNC-technology in the machine tool industry. Small
Business Economics 1997;9:361–81.
560 A.M. Sánchez, M.P. Pérez / International Journal of Project Management 20 (2002) 545–560
[34] Frame J. Quantitative indicators for evaluation of basic research
programs/projects. IEEE Transactions on Engineering Manage-
ment 1983;30:106–12.
[35] Merrifield B. Strategic, analysis, selection and management of R&D
projects. New York: American Management Association, 1977.
[36] Martı́nez A. R&D Project selection strategy: an empirical study
in Spain. R&D Management 1989;19:63–8.
[37] Krawiec F. Evaluating and selecting research projects by scoring.
Research Management 1984;27:21–5.
[38] Ellis L. Viewing R&D projects financially. Research Management
1984;27:29–34.
[39] Jackson B. Decision methods for selecting a portfolio of R&D
projects. Research Management 1983;26:21–6.
[40] Rothwell R, and Dodgson M. Innovation and size of firm, the
handbook of industrial innovation, (ed. Ch. Freeman) Aldershot,
UK: Edward Eiger, 1974. pp. 120–42.
[41] Dijk B, et al. Some new evidence on the determinants of
large and small firm innovation. Small Business Economics
1997;9:335–43.
[42] Rothwell R, Zegveld W. Innovation and the small and medium
sized firms: their role in employment and in economic change.
London: Frances Pinter, 1982.
[43] Noteboom B. Innovation and diffusion in small firms: theory and
evidence. Small Business Economics 1994;6:327–437.
[44] Rothwell R. External networking and innovation in small and
medium sized manufacturing firms in Europe. Technovation
1991;11:93–112.
[45] Ellis L. Evaluation of R&D processes: effectiveness through
measurements. Boston: Artech House, 1997.
[46] Liberatore M, Titus G. The practice of management science in
R&D project management. Management Science 1983;29:962–
74.