See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/4048675

Metrics of R&D Performance and Management of Public Research Labs

Conference Paper · December 2003

DOI: 10.1109/IEMC.2003.1252267 · Source: IEEE Xplore

CITATIONS READS

56 267

1 author:

Mario Coccia
Italian National Research Council & Yale University
316 PUBLICATIONS   8,781 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

causes of violent crime and terrorism View project

Agriculture Innovation View project

All content following this page was uploaded by Mario Coccia on 12 January 2015.

The user has requested enhancement of the downloaded file.

Jointly published by Akadémiai Kiadó, Budapest Scientometrics,
and Springer, Dordrecht Vol. 65, No. 3 (2005) 307–321

A scientometric model for the assessment of

scientific research performance within public institutes
MARIO COCCIA

National Research Council of Italy, Institute for Economic Research on Firms and Growth (Ceris-CNR),
Torino (Italy)

Nowadays, the Italian science sector is undergoing a strategic reform due to budget cuts and
there is a need for measuring and evaluating research performance of public research institutes.
This research presents a new measure to assess the scientific research performance of public
research institutes. The new model is successfully applied to 108 public research institutes
belonging to the Italian National Research Council, using data from year 2003 and displays the
laboratories with high/low performance. The results are substantially stronger and quicker to
obtain than those calculated by using conventional indicators. This model supports the policy-
makers, who must decide about the level and direction of public funding for research and
technology transfer.

Introduction

The debate on the reform of the research sector in many European countries has
recently become more important, both due to the reduction in public funding and due to
the domination of the United States and Japan in the field of new technology (SENKER,
2001; HERBST, 2004). Research is discussed both in terms of the method of public
financing, and in terms of scientific production. With their theory of the triple helix,
ETZKOWITZ & LEYDESDORFF (2000) state that nowadays universities and public
research bodies play a fundamental role in the production of inventions and innovations,
necessary for the development of a competitive industrial system, in a society that is
increasingly knowledge-based. The public sector research is, according to SENKER
(2001), defined as civil research in institutions for which the major source of funds is
public, which are in public ownership or control and which aim to disseminate the
results of their research, i.e. the defense research is excluded. The studies on these
institutions in many countries, including Italy (COCCIA & ROLFO, 2002; COCCIA, 2004),
the United Kingdom (HARRIS & KAINE, 1994; SENKER, 2001) and Finland (LUWEL et
al., 1999), show a growing interest in evaluating scientific performance (STAINER &

Received May 18, 2005

Address for correspondence:

MARIO COCCIA
National Research Council of Italy, Ceris-CNR, Collegio Carlo Alberto
Via Real Collegio n. 30, 10024 Moncalieri, Torino, Italy
E-mail: m.coccia@ceris.cnr.it

0138–9130/US $ 20.00
Copyright © 2005 Akadémiai Kiadó, Budapest
All rights reserved
M. COCCIA: Scientific research performance within public institutes

NIXON, 1997). The metrics and evaluation of scientific research (NAGPAUL & ROY,
2003; LEE, 2003) may reflect the interest of the Government in assigning clear
objectives to research laboratories so that they are managed efficiently (MORENO &
TADEPALLI, 2002) in the light of diminishing public funds. In this process of
rationalization, the Government as principal, using the terminology of the theory of
contracts, often has objectives that are in conflict with those of research institutes
(agents), above all due to imperfect information on their activities and behavior
(RADNER, 1987; HART & HOLMSTROM, 1985; LEVINTHAL, 1988). Within the research
system, the Government could have as its objective function the maximization of the
added value of less productive institutions, seeking to level out performances. This
objective can be reached by identifying high performance institutes (HPIs) and low
performance institutes (LPIs). The purpose of this research is to construct a model for
measuring the scientific research performance of public research institutes, using a
systemic approach, which considers financial, scientific and technological indices. This
model is based on a multivariate approach. The output of the model is the research
score, which is a global indicator of the scientific performance of research laboratories.
This model supports the policy-makers, who must decide about the level and direction
of public funding for research and technology transfer.
The next section describes the theoretical framework of the research, followed by
the introduction of a scientometric model. We continue with the analysis of the results
on the institutes of the largest Italian public research body, the Italian National Research
Council (CNR), using data from year 2003. Some management implications conclude
the paper.

Theory

Many studies on the measurement of research bodies’ performance consider only

the bibliometric (NARIN & HAMILTON, 1996), technometric (patents) or productive
indices (DIETZ & BOZEMAN, 2005), which clearly give partial indications on the
performance of scientific laboratories (GUAN & WANG, 2004). The economic literature
has created indicators which measure several aspects of research activities and which
are accurately synthesized through particular forms of clustering (GEISLER, 1995; 2000;
SEXTON, 1986; RUBENSTEIN & GEISLER, 1991). GEISLER (2000) states that, as clusters
of metrics are composed, at each stage of outputs, an index can be identified, which has
managerial implications. For instance, in the cluster of immediate outputs (technical
expertise, attractiveness of the Science & Technology organization), positive
normalized weights (wia) are given to each measure of each indicator so that:
n(i )
0 < wia < 1 a = 1,2,..., n(i ) and wia = 1
a =1

308 Scientometrics 65 (2005)

 M. COCCIA: Scientific research performance within public institutes

where i = number of the indicator; n(i) = number of the measures of the i-th indicator.
Thus, for each indicator the value is the sum of the weighted measures, so that the index
value for the i-th indicators is:
n (i )
Index Value(IV)= d ia wia
a =1

where wia = weight of the a-th measure of the indicator i; n(i) = number of the measures
of the i-th indicator; dia = value of a-th measure of indicator i. So, the index for each
stage of output is:
n
Index s = IV wis
i =1

where s = stage of the output; n = number of the index values; wis=0 if indicator i does
not bear on stage s. Such an overall index for stage of outputs (e.g. immediate) may also
be named factor. GEISLER (2000) points out:
• Alpha Factor = index of immediate outputs (level of technical expertise,
attractiveness of the S&T Organization);
• Beta Factor = index of intermediate outputs (level of investment in
exploitation of S&T; level of importance of S&T outcomes; climate, and
leadership);
• Gamma Factor = index of pre-ultimate outputs (investment in adoption of
S&T; structure of the industry; strategy and life cycle);
• Omega Factor = index of ultimate outputs (role and importance of S&T;
S&T level of population).
These factors are tools for managers and policy makers when evaluating Science
and Technology (S&T) at different levels. The Beta Factor, for example, describes how
firms absorb the output of science and transform it within their internal processes, while
the Omega Factor describes the total impact of science on society.
Several scholars (HARE & WYATT, 1988) measure the scientific performance of
research bodies by a series of indices, which consider all the organizational aspects of
Research & Development (R&D) bodies, from financial to technological and scientific
points of view. COCCIA (2001) evaluates the performance of public research bodies
using metrics based on the measurement of k indices, which represent the main
activities of the institutes. The model called RELEV (REsearch Laboratory EValuation)
is simple in order to bound the subjectivity. This model considers the following seven
indices for n R&D Institutes:
• = Index of public funding attributed to the research bodies = ( 1, 2, …, n);
• B = Index of self-financing = ( 1, 2, …, n );
• X = Index of personnel in training = ( 1, 2, …, n );

Scientometrics 65 (2005) 309

M. COCCIA: Scientific research performance within public institutes

• = Index of teaching activity of the researchers = ( 1, 2, …, n );

• = Index of publications with domestic diffusion = ( 1, 2, … , n);
• = Index of international publications = ( 1, 2, …, n );
• = Technometric index based on patents = ( 1, 2, …, n ).
Let i be a research institute so that i {1, 2, …, n}, let ( i, i, i, i, i, i, i ) be the
evaluation indices of i-th institute, then the REsearch Laboratory EValuation (RELEV)
function of i-th laboratory is the following combination:
1 1 1 1
relev (i) = 3 × i + × i + × i + × i +
max A maxB max X max
1 1
+ × i +2 × i + 1 se i 0; 0 se i =0
max E max

If
X1 = (1 / max ) × i;
X2 = (1 / max ) × i; and so on,
then:
RELEV (i) = 3 – X1 + X2 + X3 + X4 + X5 + 2 X6 +X7
A model (0 - 1) is applied to vector , this has the value 1 if the number of patents is
at least 1, the value 0 if there are no patents. The reason for this is to avoid penalizing
research bodies operating in the field of social or mathematical sciences which do not
produce patents, as instead do other sciences (physics, chemistry...). The RELEV Model
(COCCIA, 2001) measures R&D performance on various dimensions and gives a single
output: the R&D performance score. In fact, the values deriving from the evaluation
function RELEV = ( 1, 2, …, n) are the research performances of the laboratories.
Each i synthesizes the financial, technological, and scientific aspects of the research
institute i-th. The weakness of the RELEV model is that almost all operators have the
same weighting in the function and therefore a second RELEV model is elaborated by
COCCIA (2004), in order to improve the first one. The latter uses the discriminant
analysis with a direct method, i.e., all the output variables. The model is based on the
following discriminant function:

M = 5.178+ 1.389X1 + 1.347X2 + 1.007X3 + 0.483 X4 0.00871X5 (direct method)

where:
X1 = Self-financing (€)
X2 = Training (no. of individuals)
X3 = Teaching (no. of courses)
X4 = International publications (number)
X5 = Domestic publications (number)

Since international publications generate greater diffusion of knowledge, they are given double
weighting with respect to those published in a domestic context.

310 Scientometrics 65 (2005)

 M. COCCIA: Scientific research performance within public institutes

The application of Model II on 200 Italian public research institutes, with data from
year 2001, shows that 22.5 per cent of the total institutes are efficient (i.e. they have
high research performance). The discriminant analysis has several applications in
researches on academic institutions, such as the study carried out by HARRIS & KAINE
(1994). They investigate whether it is possible to predict the position of a researcher
group on the basis of independent variables represented by preferences and perceptions
concerning their research and environment.

Method: RELEV Model II adjusted

The research presents the RELEV Model II adjusted, which uses new samples and
the discriminant analysis with the Wilks Method. This model reduces the correlation
among some explanatory variables and represents a more reliable scientometric tool to
assess the research performance of institutes.
The first methodological step of Model II adjusted is to identify two groups of
institutes represented by:

set A or 1: High performance research institutes “HPIs”

Institutes in set A, belonging to various scientific fields, are organizations that
combine scientific excellence and high international visibility. These laboratories are
chosen because they have high output indicators per researcher (e.g. international
publications). These institutes are leaders in strategic scientific programs for the Italian
national research plan. Moreover, other characterizing elements are multidisciplinary
activities and high integration with universities, high schools, and the business world.
set B or 2: Low performance research institutes “LPIs”
Set B is made up of research institutes belonging to various scientific fields that are
included in this group because they have low indicators of output per researcher. These
institutions are characterized by a mainly administrative and bureaucratic culture within
a world scenario that emphasizes organizational and strategic management skills.

Properties of sets A and B

1. A B=

2. x A\B x A, x B.
Once the two sets or groups of the institutions are fixed, the research investigates
whether it is possible to predict the location of an institute, taken from a given
population, within one of the above subsets, A or B, on the basis of key variables.

Scientometrics 65 (2005) 311

M. COCCIA: Scientific research performance within public institutes

The following ten variables (Table 1), which concern the main inputs and outputs in
absolute numbers of public research laboratories, are considered independent variables
of the RELEV Model II adj.

Table 1. Explicative variables of the RELEV model II adj.

INPUTS
1. Public funds
2. Payroll personnel
3. Cost of labor
OUTPUTS
4. Self-financing (€) deriving from activities of technological transfer from the institute to outside users
5. Training such as degree students, PhD students, etc.
6. Teaching is the number of courses held by researchers at universities and other institutions
7. International publications are those that appear in journals listed in the Social Science Citation Index
8. Domestic publications
9. International proceedings
10. Proceedings concerning conferences with domestic diffusion

Variable 1 is the funds provided by the Government. Researchers, technicians and

administrative staff make up the payroll personnel (variable 2). Variable 3 is the basic
salary and other contributions. Self-financing (variable 4) is the revenue deriving from
research projects, scientific consulting activities, calibration, etc. Variable 5 is the
number of undergraduate and post-graduate students that study and carry out research
within the Italian National Research (CNR) laboratories. Teaching (var. 6) includes
courses carried out by CNR’s researchers. International publications (var. 7) are: i) the
number of publications in Journals classified by the Institute for Scientific Information
(ISI) in the Social Sciences Citation Index, and Arts and Humanities Citation Index; ii)
articles in other international journals, and iii) international books. Variable 8 consists
of articles in domestic journals and books published in Italy. International proceedings
(variable 9) concern international congresses and conferences, while variable 10
consists of proceedings of domestic congresses and conferences.
The discriminant function takes the following form:

(i) = 1X1 + 2X2 + 3X3 + …+ kXk

where: (i) = R&D performance of i-th Institute; i=scalar values; Xi=indices
(i= 1, 2, …, k). Another variable used in the model is a dummy variable which assumes
only two values, that is Xj = 1 (for HPIs = High Performance Institutes, grouped in set
A), Xj = 2 for LPIs = Low Performance Institutes. This variable expresses for each case
its inclusion in group or set A or B. The discriminant analysis method (MASSART et al.,
1988) uses the stepwise variable selection and adopts as selection rule the minimization
of the value of Wilks’ lambda statistic; in such cases the maximum number of steps is
20, the minimum level of tolerance is 0.001, the minimum level of F to input is 3.84

312 Scientometrics 65 (2005)

 M. COCCIA: Scientific research performance within public institutes

and the maximum level of F to remove is 2.71. The complexity and abundance of
calculations, due to the high number of variables, is overcome thanks to the application
of the SPSS statistical package, which provides all the results described and analyzed in
the following section.

Results

The RELEV model II adj. is constructed using data from the Italian National
Research Council (CNR) institutes. The CNR is the body that promotes, coordinates
and disciplines scientific research in Italy, in order to increase the scientific and
technological progress of the country. The institutional scientific activity is carried out
by 108 research institutes, operative bodies whose general aim is to do research in
relation to the programmatic objectives of the Italian government and of the European
Union.
The organizational structure of the Italian CNR is based on laboratories operating in
five scientific fields:
1. Basic Sciences, with 28 new institutes in the fields of mathematics,
physics, and chemistry;
2. Life Sciences, 33 new laboratories in the fields of medicine and biology,
agriculture and molecular biology;
3. Earth and Environmental Sciences, concerning geology, environment, and
habitat, with 10 laboratories in total;
4. Social and Human Sciences, with 19 laboratories in the fields of history,
philosophy, and philology; juridical and political sciences; economics,
sociology, and statistics; cultural heritage;
5. Finally, there is the field of Engineering and Information-Communication
Technology (ICT) Sciences, made up of 18 laboratories.
All data analyzed are taken from the official document of the Italian National
Research Council: Report, year 2003. Set A or HPI of the new methodology has 4
institutes of basic sciences, 4 institutes of life sciences, 1 laboratory of earth sciences,
and 4 institutes of engineering and ICT sciences. Set B or LPI has: 1 institute of basic
sciences, 3 institutes of life sciences, 4 institutes of social sciences and 2 institutes of
engineering and ICT sciences. The selection rule of the RELEV model II adj., i.e.
minimizing Wilks’ lambda, chooses the following five explicative variables:
a) Public funds;
b) Trainees;
c) Number of courses held by researchers in universities and other institutions;
d) International publications;
e) International proceedings.

Scientometrics 65 (2005) 313

M. COCCIA: Scientific research performance within public institutes

The prior probability of group A (or 1) is equal to 0.565, while the prior probability
of group B (or 2) is 0.435 (the total probability is 1). The canonic correlation of the
discriminant function (Wilks method) is high and equal to R 2c1=0.9596 (Table 2).

Table 2. Canonical discriminant function

Degree
1 Canonical Wilks’ of
Function Eigenvalue Chi-square Sig.
j correlation lambda freedom
(df)

1. Stepwise method of Wilks 11.622 100 0.9596 0.079227 46.906 5 0.00

The following tables (3-4-5-6) display other results.

Table 3. Standardized canonical discriminant function coefficients

Function 1
Stepwise method of Wilks

• Teaching is the number of courses held by researchers in universities

and other institutions 1.95070
• Public funds 0.57934
• Training –1.82424
• International proceedings 0.63556
• International publications 0.71517

Table 4. Structure matrix: pooled within-groups correlations between discriminating variables and canonical
discriminant function (variables ordered by size of correlation within function)

Function 1
Stepwise method of Wilks

• International proceedings 0.46143

• International publications 0.44683
• Public funds 0.42326
• Teaching is the number of courses held by researchers at universities
0.25893
and other high schools
• Domestic publications 0.23304
• Domestic proceedings 0.21166
• Trainees 0.19906
• Cost of labor 0.14065
• Payroll personnel 0.14059
• Self-financing (€) deriving from activities of technological transfer
0.08965
from the institute to outside users

314 Scientometrics 65 (2005)

 M. COCCIA: Scientific research performance within public institutes

This table shows the discriminant variables classified by size of correlation within
the function.
The unstardardized canonical discriminant function is given by:
F1 = 5.1038744 + 0.0874307(Teaching courses) + 0.00000350498206(Public funds)
0.0370964(Trainees) + 0.0171910(International proceedings) + 0.0210310(International pub.)

Table 5. Canonical discriminant function evaluated at group means (group centroids)

Function 1
Group
Stepwise method of Wilks
A=1 Y1 = centroid group A 2.85703

B=2 Y2 = centroid group B –3.71414

The cut-of-point Y0 is given by:

Y0 + Y1 2.85703 3.71414
Y0 = = = 0.428555
2 2

Table 6. Classification results

Predicted group membership
Actual group No. of cases A B

Set A (HPI Institutes) A=1 13 13 (10.00%) 0 (0%)

Set B (LPI Institutes) B=2 10 0 (0%) 10 (100.00%)
Percent of “grouped” cases correctly classified: 100.00%

Classification processing summary

23 (Unweighted) cases were processed
0 cases were excluded for missing or out-of-range group codes
0 cases had at least one missing discriminating variable
23 (Unweighted) cases were used for printed output
23 cases were written into the working file

By applying the discriminant analysis method (Wilks’ lambda) on 108 CNR

research laboratories, with data referring to year 2003 and using the above cut-of-point,
Table 7 shows that little more than 33 per cent of the Italian CNR institutes fall into the
HPIs set. Although the discriminant analysis uses absolute numbers, the results of the
different performances are confirmed by the following table, which also shows the
comparison of research performances (measured by value per researcher) between HPIs
and LPIs.

Scientometrics 65 (2005) 315

M. COCCIA: Scientific research performance within public institutes

Table 7. Discriminant analysis applied on Italian research institutes (results)

data from year 2003
Set HPI Set LPI
No. public research laboratories 108 36 72
Value per researcher
Inputs
1. Public funds (€) 7,779.27 7,124.9
2. Payroll personnel (arithmetic mean) 98.47 48.46
3. Cost of labor (€) 53,964.57 50,323.1
Outputs
4. Self-financing (€) deriving from activities of technological transfer
24,169.93 13,293.73
from the institute to outside users
5. Trainees 0.44 0.33
6. Teaching is the number of courses held by researchers in universities
0.25 0.14
and other institutions
7. International publications 1.04 0.68
8. Domestic publications 0.15 0.24
9. International proceedings 1.05 0.73
10. Domestic proceedings 0.53 0.46

The appendix shows the discriminant scores and the all-groups stacked histogram of
the analysis.

Discussion and concluding remarks

CNR’s scientific network is made up of 108 institutes subdivided according to five

scientific areas. The total number of researchers is 4,319 units (year 2001). The Italian
National Research Council can be compared to other European research institutions in
order to carry out a benchmark assessment in a global context. For instance, the Max
PlanckGesellschaft (MPG) in Germany has 80 institutes grouped into three macro-areas
and its number of employees is 3,229; the Centre National de Recherche Scientifique
(CNRS) in France has 1,640 institutes over 8 sectors and a total number of researchers
of 11,643 units; the Consejo Superior de Investigaciones Cientificas (CSIC) in Spain
has 108 institutes, grouped in 8 scientific fields. Although the size and the scientific
structure of these European research bodies are different, the following table shows that
the scientific behavior of these organizations is similar, as seen when comparing
publications and budget.

316 Scientometrics 65 (2005)

 M. COCCIA: Scientific research performance within public institutes

Table 8. Benchmark among European scientific bodies – year 2001

CNR CNRS CSIC MPG
Publication JCR 4,865 16,492 4,362 6,631
Publications per employee 0.60 0.71 0.57 0.57
Publications per researcher 1.32 1.42 1.93 2.05
Budget per publication 163,001 148,981 92,618 190,167

Source: DAST CNR (2003)

As the Italian National Research Council is similar to other European scientific

bodies (such as the French CNRS, the German Max Plank and the Spanish CSIC), the
results of this study can provide useful information to European policy makers, who
nowadays have to apply research policies and to decide about the level and direction of
public funds for research and development.
The RELEV Model II adjusted constructed for evaluating the performance of public
research laboratories is similar to those used by ALTMAN (1978) for evaluating firms
and default risk, using a series of economic and financial indices. The main issue is
whether the performance indicators and the statistical-mathematical analysis alone can
be sufficient to evaluate the performance of public research bodies and the general
health of scientific organizations (BALL & WILKINSON, 1994). The Morris Report
(1990) states that peer review, though subjective, offers opportunities for greater
analysis of the institutions with respect to the indicators of performance taken
individually. In my opinion, these two tools should be complementary. In the absence
of indicators of performance, using only elements of judgement could lead to distorted
and exaggerated information, while trusting in only one of the indices and technical
statistics could be dangerous. The performance indicators (PAGE, 1989) are good
(micro-)management tools for Research & Development Institutes, but they have the
weakness that they do not provide valid support for the scientific and research policy of
a country and, in the field of research, this is of fundamental importance (HERBST,
2004). The attempt to create a universal model has the limit that there are big
differences among scientific fields. Therefore, the future development of this research
will be based on the construction of two models to assess the research performances of
institutes: the first model will focus on natural sciences, the second one will focus on
social and human sciences.
This research show that the 36 research institutes within Italian CNR above the cut-
of-point and called HP institutes are larger in size than LPIs. Moreover, the analysis
within each research field shows the following situation: basic sciences have 53.57 per
cent HPIs and 46.43 per cent LPIs; life sciences have 30.30 per cent HPIs and 69.70 per
cent LPIs; Earth and environmental sciences have 50 per cent HPIs and 50 per cent
LPIs; technological, engineering and information sciences have 33.33 per cent HPIs and
66.67 per cent LPIs. The inclusion into set HPIs and set LPIs must not be seen as

Scientometrics 65 (2005) 317

M. COCCIA: Scientific research performance within public institutes

permanent (JOHNES, 1992), since within the Italian science sectors the gap of scientific
performance between the two sets is low. Moreover, LPIs within the Italian national
system of innovation (LUNDVALL, 1992) are characterized by poor performance due to:
a) environmental causes (reduction in public funding, changes in the technological
trajectories, etc.), b) organizational problems, such as internal conflicts in the choice of
management. These two main causes increase the so-called “X inefficiency”
(LEIBENSTEIN, 1966).
Besides the concentration measures already undertaken by the Italian Government,
another research policy to increase the performance of LPIs could be the re-location of
researchers so that they can choose in which laboratory to work according to their
scientific preferences. Moreover, to achieve the same result, it could be necessary to
perform specific actions that introduce greater incentives, for example: i) awards to
researchers for articles published in international journals (i.e. according to their
scientific quality), ii) high percentages of profit to researchers who have generated new
technology that can be transferred, iii) certain career paths and promotions linked to the
scientific recognition attained, iv) high salary to attract international scientists and to
reduce the migration of the best scholars towards foreign firms and/or universities.
HARRIS & KAINE (1994) show that, in the presence of high motivations and stimulating
environments, researchers continue to publish and are considered high performers, even
after they have reached the peak of their career.
This research confirms the concern that the Italian national system of innovation is
not working satisfactorily and that financial resources are insufficient to strengthen the
Italian scientific network in terms of production and diffusion of scientific researches
and technology in the economic system. The past and present Italian research policy,
based on cuts to public funds, is pushing towards a dangerous reduction of
competitiveness of firms and of the economic system due to low technology transfer. In
order to increase the future economic growth and wealth of the nation, the Italian
government should use the suggestions of the new theory of endogenous growth
(ROMER, 1990) i.e. increasing: 1) Gross Domestic Expenditure on R&D (GERD) as a
percentage of Gross Domestic Product (GDP); 2) GERD per capita population; 3) total
researchers per thousand labor force. However, this research also shows as the
percentage of HPIs (33 per cent) is higher than the percentage in 2001 (COCCIA, 2004),
when HPIs were 22.5 per cent of the total. This result is due to a new Italian research
policy based on the merger of research institutes within the same scientific fields, in
order to increase the economies of scale and to reduce the costs. In fact, the
reorganization of the Italian national research council (since 2002) and the reduction (as
of the mid-nineties) of the public funds assigned to research institutes is pushing the
Italian public research Institutes towards the efficient use of public resources, reducing
the X-Inefficiency. Nowadays, the CNR institutes have a different approach towards the
environment and the market, which are seen as an important financial resource

318 Scientometrics 65 (2005)

 M. COCCIA: Scientific research performance within public institutes

necessary to carry out research activities, but this phenomenon can transform research
laboratories into organizations focused on consultancy and applied research, with
negative repercussions on basic and fundamental research and, therefore, on the long-
term economic growth of the country.

I wish to thank Prof. Secondo Rolfo and the conference participants at the Indian Institute of Technology
(New Delhi) for helpful discussions and suggestions to a preliminary version of this paper. Particular thanks
go to Prof. Luca Gnan of the Università Commerciale Luigi Bocconi in Milan (Italy) for his useful tuition in
SPSS in company modelling, and to Frabrizio Tuzi (DAST-CNR) for the data set of Cnr. I am grateful to
Diego Margon and Silvana Zelli (Ceris-Cnr) for the research assistance. Thanks also to an anonymous referee
for helpful comments. Any errors are my sole responsibility.

References

ALTMAN, E. I. (1978), Financial application of discriminant analysis: a clarification, Journal of Financial and
Quantitative Analysis, 13 (1) : 185–195.
BALL, R., WILKINSON, R. (1994), The use and abuse of performance indicators in UK higher education,
Higher Education, 27 : 417–427.
COCCIA, M. (2001), A basic model for evaluating R&D performance: theory and application in Italy, R&D
Management, 31 (4) : 453–464.
COCCIA, M. (2004), Models for measuring the research performance and identifying the productivity of public
research institutes, R&D Management, 34 (3) : 267–280.
COCCIA, M., ROLFO, S. (2002), Technology transfer analysis in the Italian National Research Council,
Technovation, 22 : 291–299.
DAST CNR (2003), CNR Report, D’Anselmi (Ed.), Rome.
DIETZ J. S., BOZEMAN, B. (2005), Academic carrers, patents, and productivity: industry experience as
scientific and technical human capital, Research Policy, 34 (3) : 349–367.
ETZKOWITZ, H., LEYDESDORFF, L. (2000), The dynamics of innovation: from National Systems and
“Mode 2” to a Triple Helix of university-industry-government relations, Research Policy, 29 (2) :
109–123.
GEISLER, E. (1995), An integrated cost-performance model of research and development evaluation, Omega,
23 (3) : 281–294.
GEISLER, E. (2000), The Metrics of Science and Technology. Westport, Connecticut, London: Quorum Books.
GUAN, J., WANG, J. (2004), Evaluation and interpretation of knowledge production efficiency, Scientometrics,
59 : 131–155.
HARE, P. G., WYATT, G. J. (1988), Modelling the determinant of research output in British universities.
Research Policy, 17 : 315–28.
HARRIS, G., KAINE, G. (1994), The determinants of research performance: a study of Australian university
economists, Higher Education, 27 : 191–201.
HART, O., HOLMSTROM, B. (1985), Theory of contracts. In: T. BEWLEY (Ed.), Advances in Economic Theory.
Cambridge: Cambridge University Press.
HERBST M. (2004), Governance and Management of Research Universities: Funding and Budgeting as
Instruments of Change. Center for science and technology studies, 4, Bern, CH.
JOHNES, G. (1992), Performance indicators in higher education: A survey of recent work, Oxford Review of
Economic Policy, 8 (2) : 19–34.
LEE, C. K. (2003) A scientometric study of the research performance of the Institute of Molecular and Cell
Biology in Singapore, Scientometrics, 56 : 95–110.

Scientometrics 65 (2005) 319

M. COCCIA: Scientific research performance within public institutes

LEIBENSTEIN, H. (1966), Allocative efficiency versus «X-Efficiency», The American Economic Review,
56 : 392–415.
LEVINTHAL, D. (1988) A survey of agency models of Organizations, Journal Economic Behavior and
Organization, 153–186.
LUNDVALL, B. (1992) National Systems of Innovation, Pinter Publishers, London.
LUWEL, M., NOYONS, C. M., MOED, F. (1999), Bibliometric assessment of research performance in Flanders:
policy background and implications, R&D Management, 29 (2) : 133–141.
MASSART, D. L., VANDEGINSTE, B. G. M., DEMING, S. N., MICHOTTE, Y., KAUFMAN, L. (1988),
Chemometrics: A Textbook, Elsevier.
MORENO A. A., TADEPALLI, R. (2002), Assessing academic department efficiency at a public university.
Managerial and Decision Economics, 23 : 385–397.
Morris Report (1990), Performance Indicators, Report of a Committee Enquiry chaired by Mr. Alfred Morris,
PCFC.
NAGPAUL, P. S., ROY, S. (2003), Constructing a multi-objective measure of research performance,
Scientometrics, 56 : 383–402.
NARIN, F., HAMILTON, K. S. (1996), Bibliometric performance measures, Scientometrics, 36 : 293–310.
PAGE, E. S. (1989), Management statistics and performance indicators in British universities. Paper presented
at the 11th European Association for Institutional Research Forum. Trier, 27–30 August.
RADNER, R. (1987), Decentralization and incentives. In: T. GROVES, R. RADNER, S. REITER (Eds),
Information, Incentives and Economic Mechanisms. Oxford: Blackwell.
ROMER, P. M. (1990), Endogenous technological change, Journal of Political Economy, 98 (5) : 71–102.
RUBENSTEIN, A., GEISLER, E. (1991), Evaluating the Outputs and Impacts of R&D/Innovation, International
Journal of Technology Management, Special publication on the Role of Technology in Corporate Policy,
181–204.
SENKER, J. (2001), Changing organisation of public sector research in Europe – implications for
benchmarking human resources in RTD, Paper prepared for Human resources in RTD session of the
“The contribution of European socio-economic research to the benchmarking of RTD policies in
Europe”. Conference: Brussels, March 15–16.
SEXTON, T. R. (1986), The methodology of data envelopment analysis. In: R. H. SILKMAN (Ed.), Measuring
Efficiency: An Assessment of Data Envelopment Analysis. San Francisco: Jossey Bass.
STAINER, A., NIXON, B. (1997), Productivity and performance measurement in R&D, International Journal
of Technology Management, 13 (5-6) : 486–496.

320 Scientometrics 65 (2005)

 M. COCCIA: Scientific research performance within public institutes

Appendix
Table A1. Discriminant Scores
Case Actual Highest probability group 2nd Highest Discriminant
number group Group P (D/G) P (G/D) Group P (G/D) scores
1 1 1 0.0020 0.829 2 0.171 -0.22830
2 1 1 0.4069 1.000 2 0.000 2.02770
3 1 1 0.0546 1.000 2 0.000 4.77880
4 1 1 0.5454 1.000 2 0.000 2.25230
5 1 1 0.3606 1.000 2 0.000 3.77130
6 1 1 0.8100 1.000 2 0.000 3.09740
7 1 1 0.7513 1.000 2 0.000 3.17400
8 1 1 0.9766 1.000 2 0.000 2.82760
9 1 1 0.4608 1.000 2 0.000 3.59460
10 1 1 0.8837 1.000 2 0.000 3.00330
11 1 1 0.2987 1.000 2 0.000 1.81780
12 1 1 0.9667 1.000 2 0.000 2.89880
13 1 1 0.2045 1.000 2 0.000 4.12590
14 2 2 0.7758 1.000 1 0.000 -3.42940
15 2 2 0.5116 1.000 1 0.000 -4.37050
16 2 2 0.6681 1.000 1 0.000 -3.28540
17 2 2 0.4634 1.000 1 0.000 -2.98090
18 2 2 0.7797 1.000 1 0.000 -3.99390
19 2 2 0.5061 1.000 1 0.000 -4.37900
20 2 2 0.7763 1.000 1 0.000 -3.99830
21 2 2 0.9081 1.000 1 0.000 -3.59870
22 2 2 0.4832 1.000 1 0.000 -3.01300
23 2 2 0.7054 1.000 1 0.000 -4.09220

Figure A1. All-groups Stacked Histogram

Scientometrics 65 (2005) 321

View publication stats