Research Evaluation 21 (2012) pp. 214 doi:10.

1093/reseval/rvr001
Advance Access published on 15 February 2012

Evaluating the performance of

academic departments: an analysis
of research-related output
efficiency
Tommaso Agasisti, Giuseppe Catalano, Paolo Landoni* and
Roberto Verganti

Politecnico di Milano, Department of Management, Economics and Industrial Engineering,

Piazza Leonardo da Vinci 32, 20133 Milano, Italy
*Corresponding author. Email: paolo.landoni@polimi.it.

In this article we investigated whether academic departments do experience a trade-off among

different research outputs. More specifically, we define four types of academic research outputs:
quantity (publications); quality (citation indexes); research funds obtained through research
grants; and applied research funds obtained through external orders. Subsequently, we define
a departments performance through the concept of efficiency, namely the ability to maximize
academic research output given an amount of inputs (facilities and human resources). Using Data
Envelopment Analysis (DEA), we measure efficiency for 69 academic departments (focused on
scientific subjects) located in the Lombardy Region (Italy), benefiting from a unique data set
containing detailed information on research inputs and outputs. The empirical analysis shows
that efficiency rankings change significantly when considering different research-related outputs
and thus it highlights different research strategies among the academic departments. These dif-
ferent strategies emerge also considering jointly all four types of outputs: the academic depart-
ments focus on different outputs in order to obtain the highest overall efficiency scores. In the last
section, policy and managerial implications have been discussed.

Keywords: academic research; efciency; Data Envelopment Analysis; quality/quantity trade-off.

JEL Codes: I23; O38.

1. Motivation and objectives
In recent years, the importance of university research for
socio-economic development has been increasingly
acknowledged in the literature and at a political level
(e.g. Brennan and Teichler 2008; EU 2000 Lisbon
Declaration). At the same time, given the difcult situation
of public nances, the evaluation of research activities of
universities has become a central element in the manage-
ment and governance of single institutions and of national
systems. Specic evaluation systems have been developed
in many countries (e.g. in the USA, UK, Italy and
Australia) and they are continuously improved and
adapted to changing times (Hicks 2009).
The majority of the literature has focused on the differ-
ent ways to evaluate research outputs in terms of
publications, for instance discussing and introducing new
bibliometric indicators (e.g. the H-index) or debating on
how to develop better evaluation systems (e.g. confronting
peer reviewed-based systems with bibliometric-based
systems) (see for instance Roessner 2000; Barker 2007;
Moed 2007; Harnad 2008; Haddow and Genoni 2010;
Sala et al. 2011).
Furthermore, recent literature has focused on the impact
of scientists engagement in technology transfer activities
such as patenting, spin-off rms, and research contracts
(Van Looy et al. 2011). Even if the debate is still open,
increasing evidence and arguments points towards the
feasibility and even positive effects of combining scientic
and entrepreneurial activities (e.g. Etzkowitz et al. 1998;
Azoulay et al. 2007). In particular, many works have

The Author 2012. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com

considered the impact of patenting activities on publica-
tions quantity measured by the number of paper publica-
tions on scientic journals (see for instance Van Looy et al.
2006; Stephan et al. 2007), and on publication quality,
evaluated by citations count (e.g. Mayer 2006; Czarnitzki
et al. 2007; Breschi et al. 2008).
An important issue, in terms of policies to foster scien-
tic activities and of evaluations of these activities, deals
with the relationship between publications quality and
quantity on one hand and on the ability to attract public
and private funds on the other hand. These themes have
received a more limited attention in the literature, but are
pivotal to the political debate. For instance, if national
policies and evaluation systems focus exclusively on the
quantitative aspect of publishing, a shift in scientists be-
haviour may result, e.g. with researchers encouraged to
conduct studies more short-term oriented and easy to
publish in international journals, instead of more funda-
mental, high-quality, long-term, and riskier activities. As a
matter of facts, the relationship between publication
quality and quantity, the impact of policies and evalu-
ations and the related theme of quality indicators (e.g. cit-
ations) is still under debate (see for instance Aksnes and
Rip 2009).
A similar reasoning can be applied to the other two
research-related outputs, namely research funds obtained
through research grants and applied research funds
obtained through external orders.
In this article, we aim at contributing to the discussion
regarding the potential correlations and trade-offs among
different research outputs, by focusing on the evaluation of
academic departments efciency. We do not consider the
antecedents of these outputs or the factors inuencing
them, but we focus on the output themselves and the ef-
ciency of academic departments.
The main aim of the article is to compare performance
scores for these departments in terms of different
research-related outputs; such results could be important
for scholars, policy-makers (allowing to make judgements
and eventually resource allocation decisions) and institu-
tions managers (benchmarking exercises and internal
evaluations can be conducted).
Considering efciency as the maximum output obtain-
able employing a given set of inputs (output-oriented
setting) the main research questions of this work are: Do
efciency scores change when considering different research
outputs types? And if yes, how?
These research questions aim at verifying if academic
departments that excel on a research-related output excel
also in one or more of the others or whether different
research strategies and foci exist among the academic
departments. Furthermore the study is aimed at verifying
the presence of trade-offs or correlations among these
different outputs in order to shed some light on the
different strategies of academic departments. The provi-
sion of empirical evidence on this issue is important for
Evaluating the performance of academic departments . 3
methodological reasons: if it clearly emerges that efciency
varies accordingly to the research-related output con-
sidered, then the same related outputs cannot be used as
alternatives, as they are not equivalent in explaining
research performances. For instance, previous studies
claimed that the use of research publications or grants
can be used interchangeably as a proxy for research
output, but if it is found that there is a trade-off among
these dimensions, then empirical analyses should treat
them separately.
The analyses considers both the efciency rankings
obtained using only one output at time and the overall
efciency that includes all four research-related outputs,
as it will be further illustrated in the methodological
section.
The research questions are investigated empirically using
data of (disciplinary homogenous) academic departments
located in an Italian Region, namely Lombardy. Our
sample includes 69 departments on science, technology,
and medicine. The choice of Lombardy is due to data
availability (see next section) and to the relevance of
universities located in this Region with respect to the
whole national HE sector. However, we tried to derive
wider conclusions that can apply also to different HE
systems and can stimulate more general reections.
The article is organized as follows. The next section
contains a review of the recent relevant literature in this
eld. The third section illustrates the methodology and
data. The fourth section shows the results, while the fth
discusses and concludes.
2. Literature review
2.1 Contributions on research-related outputs
The literature on the topic of research-related output is
wide and diverse. One of the main themes currently under
debate regards the relationship between publication
quality and quantity, considering in particular the
impact of policies and evaluations (Aksnes and Rip
2009). Several studies warn about national policies and
evaluation systems too focused on one aspect or
another, as a shift in scientists behaviour may result.
For instance a longitudinal analysis (20 years) of UK
science conducted by Moed (2008) has shown three
distinct scientists responses to the principal evaluation
criteria applied in the UK Research Assessment
Exercise (RAE), and in particular two shifts of research-
ers from quality to quantity. Some authors (e.g. Hayati
and Ebrahimy 2009) have argued that there could be a
positive and meaningful correlation among qualitative
and quantitative criteria in the scholarly scientic publi-
cations. Others (e.g. Dewett and Denisi 2004) have
explored the relationship between quality and quantity
in terms of scientic reputation, suggesting quality,
4 . T. Agasisti et al.
creativity in particular, is more important than quantity
for the accumulation of reputation.
Recent studies have focused on identifying determinants
and hampering factors of the scientic output of research-
ers. For example, referring to French and Italian cases,
Lissoni et al. (2009) proved that scientic outputs are
inuenced by factors like the size and the nature of
projects (national or international, collaborative or not
collaborative), authors age and gender.
As previously noted, in terms of determinants and
hampering factors a vast literature has regarded in particu-
lar the impact of involvement in patenting activities.
With regard to publication quantity, the majority of the
authors (e.g. Breschi et al. 2005; Van Looy et al. 2006;
Stephan et al. 2007) state that academic patent inventors
publish more than their solely publishing, non-inventing
peers. Conversely, in terms of quality the results are
more discordant. Many authors found a strong positive
effect, but some scholars were led by their ndings to
identify a weak positive correlation among academic pa-
tenting and citations count (see Azoulay et al. 2006), or
even a negative one (see Fabrizio and Di Minin 2008).
Among the research-related outputs increasing attention
is being devoted to the ability of attracting research grants
and external orders. These capacities are increasingly con-
sidered as objectives of the universities and as proxies of
the quality and the relevance of the research performed. In
particular, faced with tight budget constraints, universities
allow more time of their researchers for income-generating
activities such as applied research or consultancy (see for
instance Arvanitis et al. 2003; Beath et al. 2003) and many
governments foster researchers collaboration with rms
and external institutions (e.g. Coccia and Rolfo 2008).
Some authors (e.g. Kroll and Schiller 2010) even suggest
that applied research institutes (including universities)
should be nanced mainly by rms than through govern-
ment funds.
Given the changes in universities funding, many
scholars have started to focus on the relation between sci-
entic research and the different sources of funding (see for
instance Buccola et al. 2009). In particular, they have been
interested in evaluating the inuence of governmental and
industrial funds both on the technological output of
applied research (e.g. Dai et al. 2005; Chang et al. 2006),
and on research productivity. In terms of impacts on
research productivity, results are not always concordant.
Gulbrandsen and Smebys (2005) ndings suggest a correl-
ation between private funding and research performance,
in terms of publications, whereas Goldfarb (2008) found
that industry-funded researchers experience a decrease in
publications. Manjarres et al. (2009) reached a middle-way
conclusion, stating that the effects of R&D contracts with
industry and academic research activity on scientic pro-
duction are synergistic: there is a synergy effect more than
a substitution effect.
2.2 Empirical studies about the efficiency of academic
research using Data Envelopment Analysis
A second important stream of the literature concerns the
use of quantitative approaches to measure the efciency of
academic units.
The empirical international literature about universities
efciency has been largely focused at department-level,
with the aim (i) to compare departments efciency
across universities (in homogenous elds) or (ii) to
compare departments efciency and productivity within
the same HE institution.
Among the rst group of studies, a lot of attention has
been devoted to the Economics Departments. Johnes and
Johnes (1993, 1995) measured the research performance of
UK economics departments, by using staff numbers as
inputs and publication counts and grants as outputs.
These two contributions focus exclusively on the research
dimension of the universities work, and not provide
insights about the teaching dimension (this is the same
choice adopted in our article). Madden et al. (1997)
studied the efciency of economics departments in 29
Australian universities. Inputs are teaching and research
staff, outputs are graduates and publications (for
teaching and research, respectively). The aim was to
verify if policies promoted by the governments impacted
on the productivity. Thursby (2000) analysed the
Economics Research Departments efciency in the USA.
He used publication and citation data, and number of
PhDs as outputs, and faculty size, full professors,
graduate students, federal grants, library acquisitions,
and a dummy for private ownership as inputs, nding
that the average level of efciency in the USA is high.
Moreno and Tadepalli (2002) focused on 42 academic
units distributed across 7 colleges in the USA. They used
staff, nancial resources and proxies for structures (e.g.
building space) as inputs, and number of students, FTE
enrolments and grant awards as outputs. Their particular
background is strongly managerial, and the analysis
results have been used as a planning tool.
Moreover, there are studies focused on the performance
of academic units within a single university. Koksal and
Nalcaci (2006) derived efciency scores for 14 university
departments at the Middle East Technical University
(METU) in Turkey, by using staff salaries, previous
research quality and number of students as inputs, and
number of publications, graduates, and teaching indicators
as outputs. Gimenez and Martinez (2006) focused on cost
efciency of the departments at the Autonomous
University of Barcelona. The units of analysis are 42 de-
partments, and again the focus is on both teaching and
research activities. The chosen inputs are: expenditure on
temporary hired teaching and research staff, operational
expenditures, expenditures on tenure staff. To measure the
outputs, they employed the results of a government evalu-
ation process of research, teaching load (measured through

credits granted), and students opinions (through
questionnaires).
In Italy, analyses about the efciency of Italian higher
education have been mainly conducted at institution-level
(thus, not considering separately the structures devoted to
teaching and research). Agasisti and Dal Bianco (2006)
studied the efciency of 58 Italian public universities, by
using number of staff, nancial resources, and students as
inputs, and graduates and research grants as outputs.
Later, Agasisti and Salerno (2007) analysed the cost ef-
ciency of another sample of Italian universities, by adding
some qualitative measures for inputs and outputs.
Bonaccorsi et al. (2006) used a robust non-parametric tech-
nique (Daraio and Simar 2007) to take in account external
conditions affecting the productivity of universities. The
inputs and outputs adopted are quite similar to previous
studies (that is, staff numbers, expenses, and spaces as
inputs, and publications and graduates as outputs), but
they also include some external factors: university size,
number of departments within the university, percentage
of private funding, number of curricula activated. There
are also two contributions at departmental level (Pesenti
and Ukovich 1996; Rizzi 1999) that use a similar set of
variables, and focus both on teaching and research
outputs, but they were developed by using departments
within a university as the unit of analysis. The main
scope for these contributions is to provide information
and suggestion for allocating resources within a university.
To sum up, the empirical literature seems to suggest
that, to analyse research productivity of universities/
departments:
. inputs were classied under three main groups: human
resources, nancial resources, and structures (e.g.
facilities, buildings, etc.) and
. outputs have been grouped into publications and nan-
cial support (e.g. grants).
In this respect, our article is innovative because it
explores potential trade-offs among different types of
research outputs (grants, publications, etc.). As discussed
above, much of the previous (empirical) literature
employed one measure for research output at time,
without testing the effects of considering different vari-
ables simultaneously. Moreover, this is the rst attempt,
in Italy, to compare the performance not of universities as
a whole, but of single departments in a specic eld
(namely, scientic disciplines).
3. Methodology and data
3.1 The methodological approach
Data Envelopment Analysis (DEA) is a deterministic
frontier technique, whose aim is to model the production
process of homogenous organizations, to derive an ef-
ciency frontier. It does not require a priori functional
Evaluating the performance of academic departments . 5
specication to mix input and output, and it is based on
a linear programming technique, introduced by the
seminal work of Charnes et al. (1978). DEA specically
deals with the efciency problem of complex organizations
that produce many outputs using several inputs at a time.
The explanatory power of traditional index measures
based on input/output ratio is seriously affected in these
situations, as they are not able to capture the multi-input, -
output dimension of the problem. Because of this DEA
advantage (ability to employ multiple-inputs and -
outputs) it has been widely used in the empirical literature
about HE efciency (Johnes 2006).
In this article, we only briey discuss some characteris-
tics of DEA, while Cooper et al. (2006) and Zhu (2003)
provide details.
In a DEA model, the notion of efciency is that of tech-
nical efciency, dened as the relative ability of each
Decision Making Unit (DMU, in our case departments)
in producing outputs, where the term relative means that
each organization is compared with any other homoge-
neous unit. The choice of a set of weights that combine
several outputs and several inputs is the core of DEA
analysis. Indeed, the weights are chosen by DEA technical
algorithm: in this sense, DEA could be represented by a
linear programming technique where each DMU tries to
maximize the efciency ratio (output over inputs) choosing
the best set of weights. In other words, the weights can be
interpreted as the relevance of each input or output in
determining the efciency score of the organization. By
looking at weights (as derived by the DEA technique), it
is possible to understand how the single inputs and outputs
contribute to the overall organizations performance.
There are two ways to deal with a DEA formulation: (i)
input oriented (the ability to minimize inputs when outputs
are given), and (ii) output oriented (maximization of
outputs given a certain bundle of inputs). In this article,
we adopt this latter approach, as the typical problem for
an academic department is to maximize outputs (e.g. pub-
lications, grants, etc.) given its inputs (e.g. research staff,
laboratories, etc.). Following Johnes (2006), the mathem-
atical formulation is:
X
s X
m
max k + e sr +e si 1
r 1 i1
Subject to
X
n
k yrk  j yrj+sr 0, r 1, . . . , s
j1
X
n
xik  j xij  si 0 i 1, . . . , m
j1
Here, there are s outputs and m inputs. yrk is the output r
produced by the k-th unit. xik is the amount of input i used
by the same k-th unit. Lastly, sr is the output slack, and sj
the input slack. The slacks represent the amount of input
6 . T. Agasisti et al.
that can be reduced (or the amount of output that can be
incremented) to obtain an efcient performance; they are
calculated with respect to a benchmark organization in the
sample. By construction, efcient organizations slacks are
equal to zero.
DEA mathematical formulation can deal with both
constant returns to scale (CRS) and variable returns
(VRS). The seminal work of Charnes et al. (1978)
introduced a constant return to scale (CRS) model where
DMU dimension has no importance in dening efciency
performance. DMUs face the same efciency frontier, in-
dependently of their relative size. Obviously, one can cast
doubts on the comparability of small and large units.
Larger units exploit common inputs to produce different
outputs, whereas smaller ones benet from substantial ad-
vantages in organizing activities. The BCC model, de-
veloped by Banker et al. (1984) introduces in DEA
modelling the dimension factor simply adding the follow-
ing constraint to the Equation (1):
X
n Bergamo, Brescia, Pavia, and Varese, which is explained
j 1 2 by the localization of universities in those cities. We built
j1
3.2 Data
In this article, we used a unique data set that consists of
input and output variables about research and technology
transfer activities in scientic university departments in
Lombardy Region (Italy). This data set has two main ad-
vantages: (i) departments can be treated as homogenous
(in terms of subjects and disciplines but also in terms of
administrative personnel, procedures, etc.) and (ii) the
external conditions (e.g. industrial development of the
area) are similar for all the academic departments.
This data set is based on information coming from
QuESTIO system, a mapping of scientic Research and
Technological Transfer activities developed in Lombardy
Region (www.questio.it). We attempted to take advantage
of the richness of the data set, which includes much infor-
mation about the research outputs produced by the uni-
versity departments. The QuESTIO system, launched in
2004 by the Regional Government, was intended to be a
transparency and incentive tool for the Research and
Technological Transfer market by reducing information
unevenness often binding cooperation among producers
and users of knowledge. The system pursues such scopes
by collecting, updating, and publishing nancial, organiza-
tional, and structural characteristics of research organiza-
tions and Knowledge Intensive Business Services (KIBS) in
Lombardy.
The data set has some specic criteria that are useful for
our analyses. First, the system is focused on every structure
where such activities are carried out, even where that struc-
ture is a part of a wider organization. Second, these struc-
tures should have signicant management autonomy so
that accountancy, ascribing solely to that specic ofce
under evaluation, shall be determinable. The rst criterion
enables the regional administration to collect information
on every single department, being acknowledged that de-
partments belonging to the same university can differ sig-
nicantly in terms of performances. As a consequence
university departments, as well as sections and institute
of National Research Council or R&D unit belonging to
a Company, provide data making solely reference to their
specic resources and outputs. The second criterion
assures that each structure included in the data set is re-
sponsible (to some extent) for its strategy and perform-
ances thus justifying application of DEA technique.
University departments are main actors in the QuESTIO
system; as a matter of fact they are 105 out of 238 struc-
tures mapped.
The structures are spread within the regional territory: a
geographical analysis shows a concentration in the
Province of Milan, and a signicant presence in
our data set by considering those university departments
that provided information for the year 2008. This choice
has been due to the limited availability of indicators about
research quality (citation indexes) that were not derived for
the period 20047.
We selected the following variables:
3.2.1 Inputs. (1) Laboratories (Lab)
This variable considers the areas (measured in square
meters) devoted to laboratories belonging to or accessible
by the department. This information provides a dimen-
sional data related to the most distinctive and
characterizing areas equipped for scientic research.
(2) High-Qualied Human Resources (Staff)
It refers to persons that perform research and technol-
ogy transfer activities, regardless of his/her qualications
that are mainly professors, associated professors, perman-
ent researchers, post-Doc scholarships, and PhD students.
It includes employees dedicated to activities in support of
research, like: certication, tests and measurements, pa-
tenting and commercialization, nancial assistance,
partners search, technologies monitoring, events organiza-
tion, editing of technical-scientic spreading publications.
We classied outputs into four categories: quantity
(number of publications), quality (citation indexes),
grants from research bodies, and applied research
through externally funded orders (e.g. companies, public,
and private organizations, etc.).
3.2.2 Outputs. (1) Yearly number of publications (Pub)
This variable reports yearly articles signed by de-
partments human resources on the international
 Evaluating the performance of academic departments . 7

scientic press. This is the traditional and widespread Table 1. Inputs and outputs: an overview
variable used to measure the output of research activities.
(2) Citations per article (Cit) Variable Description Type
name
Average number of citations for each article published
by departments human resources.
(3) H-index (H) Lab Laboratories (m2) Input
Staff High-qualied research staff Input
It is one of the most recent (and widespread) indicators
Pub Yearly number of publications Output
in bibliometrics. A department has index h if h of P papers Cit Citation index: number of citations per article Output
has at least h citations each, and the other (P  h) papers H H-index Output
have at most h citations each. Grants_1 Revenues from national and regional research Output
(4) Research funded through regional or national grants grants
Grants_2 Revenues from international research grants Output
(Grants 1)
Ext Revenues from orders (external-funded research) Output
Accomplished activities nanced with regional or
national calls for bids. This variable denotes the ability
of the department to compete with other research organ-
izations in getting public-funded research, thus measure
indirectly the quality of research activities with reference Table 2. Correlations among outputs
to national boundaries. It is measured through an account-
ing variable standardized by the number of researchers Pub Cit H Grants 1 Grants 2 ext
(euro per researcher). It is a self-reported measure
included in the questionnaire lled by each department. Pub 1
(5) Research funded through international grants Cit 0.797 1
H 0.205 0.577 1
(Grants 2)
Grants 1 0.367 0.201 0.060 1
Accomplished activities nanced shows the capacity of Grants 2 0.274 0.320 0.122 0.582 1
drawing on international funding, like the European Ext 0.143 0.130 0.253 0.530 0.294 1
projects, resulting by working with third Countries organ-
izations. It reects department downstream international-
ization and it is a sign of the quality and competences of
the centre competing at international level. It is measured Table 3. Descriptive statistics: input and outputs
through an accounting variable standardized by the
number of researchers (euro per researcher). It is a Variable Obs Mean SD Min Max
self-reported measure included in the questionnaire lled
by each department. Lab 70 2,369.9 7,146.7 0.0 60,180.0
(6) Applied research through externally funded orders Staff 70 96.6 77.7 8.0 434.0
Grants 1 70 123,246.3 264,688.3 0.0 1,805,922.0
(ext)
Grants 2 70 493,309.7 566,339.1 0.0 3,039,361.0
This variable measures how far research and technology H 70 14.3 8.5 1.0 40.0
transfer activities carried out in a department are Ext 70 411,208.1 706,985.0 0.0 3,333,362.0
committed directly by companies or other institutions, Cit 70 6.4 3.4 0.0 15.0
thus indicating direct recognition of the value/quality of Pub 70 32.9 32.3 0.0 188.0
its know-how. It is measured through an accounting
variable standardized by the number of researchers (euro
per researcher). It is a self-reported measure included in the
questionnaire lled by each department. efciency scores modify when considering the four
A synopsis of the variables employed in the present research-related outputs separately (different denitions
study is illustrated in Table 1. Table 2 shows the correl- of research output). The overall model (including all the
ations among outputs; interestingly, while there is correl- research output types simultaneously) has been employed
ation between quality and quantity of publications, much to see whether different departments strategies emerged in
less accordance is between publications and externally the composition of the research output portfolio.
funded research (in some cases, there is even inverse An important methodological note is due here. As
relation). Table 3 contains the descriptive statistics for widely known, signicant differences exist across different
the inputs and outputs employed in the study. scientic elds in terms of publication behaviours (e.g.
At the end, we have complete data for 69 academic de- number of authors per publication, number of publications
partments, while 35 of them have some missing data per year, and average number of citations). Because of this
injuring the analysis (thus, we excluded them from our reason, we rescaled the data according to the differences
sample). We ran ve different DEA models (Table 4). in the elds. This approach consists of several subse-
The rst four models have been used to investigate how quent steps. In the rst stage, ve different groups of
8 . T. Agasisti et al.

departments were created considering the different scien- among departments. From a theoretical point of view,
tic elds. The scientic groups were created and named as there were two approaches to be adopted: (i) to run differ-
follows: ent efciency analyses for each subgroup, or (ii) to re-scale
the data set by considering the averages for each subgroup
. FIS (8 departments): it comprises Mathematics and
of departments (i.e. correcting for the discipline-specic
Physics;
averages). The second approach was preferred because it
. BIO (16): Biology and Life sciences;
allows for maintaining a larger sample that has beneted
. CHIM (11) Chemistry;
the quantitative analysis. Mathematically, every inputs and
. MED (17): Medicine; and
outputs have been re-calculated as a ratio with the
. ING (18): Engineering.
discipline-specic mean:
The descriptive statistics provided by discipline conrms
Oij
strong heterogeneity (Table 5): for instance, the average Oij 3
citation rate for FIS departments is almost double than Om
j
the average citation rate for ING ones. where Oij is the output of the i-th department (belonging to
In the second stage, the data set has been corrected the j-th discipline) used in the analysis; Oij is the observed
to take into account the discipline-specic differences output and Om j is the mean value observed in the j-th dis-
ciplines sample of departments.
Finally, the re-scaled data set has been used for the em-
Table 4. Models specication
pirical analyses through Data Enveloped Analysis. This
way, each departments performance is depurated by
Model Description Indicators the effect due to discipline-specic patterns.

A1 Quantity Pub
A2 Quality Cit, H 4. Results
A3 Research grants Grants 1, Grants 2
In this section, we illustrate the main results obtained
A4 Applied external Ext
B Overall Pub, Cit, H, Grants 1, Grants 2, Ext through the empirical analysis, while the next will discuss
the results in terms of policy implications.

Table 5. Descriptive statistics, by eld

Lab Staff Grants 1 Grants 2 Ext Pub H Cit

FIS Mathematics and Physics (8)

Mean (SD) 836.3 (676.7) 103.5 (63.6) 57,795.8 (80,213.2) 407,630.7 (431,788.0) 148,555.8 (336,676.1) 72.6 (53.1) 23.9 (11.3) 7.3 (3.8)
Median 746.0 96.5 26,242.5 217,557.2 24,404.5 70.0 24.0 7.7
Min 36.6 33.0 1.0 77,792.0 1.0 5.0 1.0 0.3
Max 2,000.0 219.0 229,370.0 1,250,000.0 978,209.0 188.0 40.0 13.6
BIO Biology and Life Sciences (16)
Mean (SD) 1,557.8 (1,630.9) 80.5 (60.1) 140,298.5 (217,425.8) 779,903.9 (801,049.3) 187,923.1 (149,119.5) 25.7 (25.9) 13.8 (7.8) 6.9 (3.4)
Median 968.0 89.0 52,512.5 481,342.5 168,534.8 18.0 13.5 6.9
Min 1.0 8.0 1.0 1.0 1.0 1.0 3.0 1.0
Max 6,000.0 224.0 718,810.0 3,039,361.0 496,812.7 77.0 33.0 12.8
CHIM Chemistry (11)
Mean (SD) 2,570.0 (2,238.9) 81.3 (64.9) 95,108.3 (164,403.6) 431,061.5 (428,644.6) 426,595.0 (860,837.6) 36.5 (22.4) 16.4 (5.9) 6.3 (3.0)
Median 1,750.0 64.0 1.0 261,145.0 131,525.0 29.5 16.5 6.0
Min 550.0 34.0 1.0 44,000.0 1.0 16.0 7.0 2.2
Max 8,000.0 253.0 494,375.5 1,401,122.0 2,854,450.4 87.0 27.0 13.4
MED Medicine (17)
Mean (SD) 1,221.8 (956.8) 94.9 (95.1) 54,304.7 (115,173.0) 383,769.3 (346,131.6) 200,864.5 (310,712.6) 31.6 (21.1) 16.5 (7.0) 7.9 (3.1)
Median 1,200.0 69.0 1.0 251,643.0 98,762.0 34.0 15.0 7.7
Min 50.0 29.0 1.0 1.0 1.0 1.0 5.0 4.0
Max 4,000.0 434.0 472,000.0 1,141,346.1 1,193,846.0 80.0 28.0 13.7
ING Engineering (18)
Mean (SD) 1,534.7 (926.9) 122.2 (86.3) 224,770.2 (440,329.1) 439,433.7 (582,275.1) 937,419.4 (1,026,626.9) 22.6 (28.6) 7.9 (4.7) 4.3 (3.2)
Median 1,200.0 96.0 37,967.5 187,201.5 541,754.0 11.5 7.0 3.0
Min 200.0 32.5 1.0 1.0 3,750.0 1.0 1.0 0.8
Max 3,069.3 386.0 1,805,922.0 2,282,734.0 3,333,362.0 116.0 21.0 15.1
 Evaluating the performance of academic departments . 9

Table 6. Efciency scores: summary statistics

Statistics Model A1quantity Model A2quality Model A3grants Model A4applied Model Boverall

Mean (SD) 0.485 (0.290) 0.634 (0.231) 0.462 (0.310) 0.175 (0.265) 0.777 (0.209)
Median 0.484 0.617 0.363 0.077 0.788
Min 0.016 0.067 0.040 0.000 0.312
Max 1.000 1.000 1.000 1.000 1.000

Notes: All the models have been derived through DEA Excel Solver provided by Zhu (2003)in its Excel2007 version. Models characteristics: output-orientation; Variable
Returns to Scale.

Table 7. Efciency scores: correlations

Model A1 A2 A3 A4

A1 1.000
A2 0.558 1.000
A3 0.039 0.333 1.000
A4 0.178 0.111 0.314 1.000

Table 6 tabulates the descriptive statistics for the

variable returns to scale (VRS) efciency scores. As
usual in DEA analysis, the overall model shows higher
mean scores, as many variables are included in the
analysisthus, each department is evaluated under more
dimensions, and some of them can determine its efciency.
However, what clearly emerges is that the model quality
has the relative higher efciency scores, while the model
applied external has the lower ones. It means that the
departments are more different in terms of ability in at-
tracting external funds than in terms of publication
citations.
As it can be noted academic departments with high ef-
ciency levels in one output can have signicantly lower
efciency levels on other outputs thus already answering to
our research questions.
These differences are clearly conrmed and further
detailed analysing the (Pearson) correlations among the
different models (Table 7). First, quality and quantity of
research are signicantly related (corr. >0.55), that is the
ability to publish more is associated with higher per-capita
citations.
On the contrary the correlation of quality and quantity
of publications is, instead, very lowly associated with the
ability to attract more resources (both grants or external
funds): indeed, the correlations are all very low (corr. <0.1
for the quantity model against A3 and A4; corr. <0.34 for
the quality model against A3 and A4). The evidence
suggests that those departments that have the highest
levels of efciency in terms of publications or citations
do not necessarily have also the highest levels of efciency
in the ability to attract resources.
Figure 1. The efciency scores for the overall model: a
graphical distribution.
Finally, it is also important to underline that also
grants and applied external models are poorly
related (corr. <0.32), suggesting that there is a trade-off
with respect to the specic type of applied research con-
ducted in the department.
A further conrmation of this interpretation comes
when looking at some details of the model overall. The
Figure 1 shows graphically the distribution of the ef-
ciency scores, denoting that many departments emerge as
efcient. It preliminarily suggests that, when considering
the four outputs simultaneously, the different departments
are heterogeneous in the way each of them combine
outputs to maximize its own productivity.
To analytically explore this evidence, it is useful to look
at the weights associated to each output. Indeed, by
analysing the weights, we can understand on which
output each department relies upon to determine its own
efciency. In other words, the weights (as calculated by the
DEA algorithm) provide indirect information about the
strategy pursued by each departmentthat is, which
output is more important to determine its performance.
From Table A1, which illustrates the weights for all the
departments, it is possible to understand how efcient
units can have a different strategy (thus, underlying
10 . T. Agasisti et al.

Table 8. Outputs virtual weights for the overall model

Grants 1 (%) Grants 2 (%) Ext (%) Pub (%) H (%) Cit (%)

Mean (SD) 9.73 (18.51) 17.73 (26.01) 7.46 (13.18) 23.79 (35.65) 22.70 (31.26) 18.58 (24.98)
Median 0.00 0.45 1.03 0.00 4.00 0.00
Min 0.00 0.00 0.00 0.00 0.00 0.00
Max 100.00 93.80 55.28 100.00 100.00 98.56

Table 9. Outputs virtual weights for the overall model, by discipline

5. Discussion and concluding remarks
Grants 1 Grants 2 Ext Pub H Cit The present study focused on a widely debated topic about
(%) (%) (%) (%) (%) (%) the trade-offs among different research-related outputs.
More specically, we dened four types of academic
FIS research outputs: (i) publications, (ii) citation indexes,
Mean 3.85 26.93 14.98 4.72 33.23 16.29 (iii) research grants, and (iv) applied research funds
Median 0.00 17.62 0.00 0.00 25.31 14.92 obtained through external orders. We employed DEA to
BIO
Mean 15.13 20.91 2.26 19.80 14.57 27.33
explore the efciency of academic departments by con-
Median 7.17 11.69 0.00 0.00 0.00 31.63 sidering different specications of research outputs. Our
CHIM results suggest that, while there is a substantial accordance
Mean 3.97 7.66 8.15 39.06 19.81 21.36 between analyses considering research quantity and quality
Median 0.00 0.00 0.66 14.65 0.00 7.17 (publications and citations, respectively), the departments
MED
Mean 13.28 20.07 6.14 35.16 15.43 9.92
more focused on these dimensions are less involved in
Median 0.00 13.41 1.77 20.32 0.00 0.00 activities aimed to attract nancial resources (grants and
ING orders). Moreover, when considering all the four research
Mean 8.23 15.38 9.21 12.66 34.53 19.99 output simultaneously, it turns out that many departments
Median 0.00 0.00 5.31 0.00 17.49 0.00
focus mainly on publications to determine their perform-
ance, while the group of departments relying upon
research grants and external orders is quite limited. This
departments heterogeneity). For instance, Department 2 is last nding conrms a weakness of many Italian
efcient because it has a high citation score (the weight universities in attracting research funds from external
associated to it is equal to 65%) the Department 48 is sources.
efcient because of its ability to attract international- Our article has some clear policy-making implications.
funded grants (the references virtual weight is 44%). Indeed, the results can be used to analyse how departments
Moreover, there are departments that are efcient by behave with respect to the different research-related
balancing their activities. outputs (for instance, publications versus grants). When
The analysis of the weights can be particularly useful for the policy-maker sets incentives, the results can be
the policy maker and the evaluator of the system, as it is utilized to assess whether the incentives changed the de-
possible to individuate strengths and weaknesses, as well as partments behaviours. A possible application, in this
priorities for each department. At the same time, some context, concerns the RAE recently promoted by the
patterns can be observed by analysing the average data Italian Ministry of Education (Decree n.8, 19 March
(Table 8). Indeed, among the 69 academic departments, 2010). This exercise will reward mainly publications, and
the majority concentrates its strengths on publication it is likely to produce an incentive to (i) publish more and
numbers and H-index, while the ability in attracting (ii) in more relevant journals (as bibliometric measures will
research grants is less relevant in determining efciency be included). The methodology employed in our article
scores. could be used, after the Exercise, to analyse whether
Finally, there are differences among the disciplines that such shift towards publications will occur or not.
can be appreciated by an analysis of the data disaggregated This argument suggests an institutional note regarding
by eld (Table 9). For instance, from the analysis emerges data availability. To fully exploit the benets of our meth-
that the Physics departments focus on average on the odology, the Ministry of Education should develop a
H-index (average virtual weight is around 33%), while complete data set about publication, citation indexes,
Chemistry departments rely more on the number of pub- grants, etc. for each Italian academic department; such
lication (virtual weights mean is around 40%). The table source of information is not available today. This also
allows to observe wide differences on this ground. explains why our article refers only to scientic

departments of universities located in the Lombardy
Regionas it collected the complete data of departments
working in its competence area. The establishment of a
national up-to-date database on the research products
carried out by single departments would allow major im-
provements in the research about the determinants of
research productivity.
Our results can also have managerial implications, in
particular through the use of the output-slacks. These
are dened as the difference between (i) the observed
value of each output and (ii) the output value that
should be observed if the department was efcient (pro-
jecting the department on the efciency frontier). As a
consequence, the output slacks are equal to zero for all
the efcient units. This information could be useful for
managerial purpose and in particular to assess a depart-
ment relative positioning. For instance, reading Table A2
(which reports the standardized output slacks for the
overall model), each departments manager can know
exactly how much grants from government must
improve, or external funded research must increase, to
allow its department reaching the efciency (operationally,
by multiplying the value reported in the Appendix 1 for
each departments input/output for the corresponding
eld-specic average). Thus, each department can have a
guideline that quanties the distance from the efciency
frontier not only in terms of scores (reected in the ef-
ciency scores) but also in terms of outputs expansion/
inputs reduction. Such information would improve the
set of data and information available to build internal
control systems in HE institutions.
The article has three main limitations, which suggest a
potential agenda for further research. First, only scientic
departments are included in the QuESTIO system. Along
with the eventual expansion of the database, to include
other sectors (e.g. humanities, social sciences, etc.) the
empirical analysis will shed more light also on these new
groups of departments. Second, we did not fully exploit
differences among different disciplines (e.g. chemistry
versus engineering). Accounting for this heterogeneity
will help us to further rene our ndings. Third,
we focused only on the Lombardy region (Italy), an ex-
tension of the analyses to other regions can be interesting
to check the validity of the results in other contexts.
Finally, it could be interesting to explore a longitudinal
set of data to understand how efciency scores change
over time.
However, our article is noteworthy because, for the rst
time, the empirical analysis explicitly modelled four differ-
ent research-related outputs, and accounted for potential
trade-offs and complementarities among them. Future
research will help to go deeper in the empirical exploration
of the results, which could be very important for both
research policies and effective management of academic
departments.
Evaluating the performance of academic departments . 11
Acknowledgements
We grateful acknowledge the Eupolis regional agency for
the data and the Editor and two anonymous referees for
their comments. The usual disclaimers apply.
References
Agasisti, T. and Dal Bianco, A. (2006) Data Envelopment
Analysis of the Italian University System: Theoretical Issues
and Policy Implications, International Journal of Business
Performance Management, 8/4: 344367.
Agasisti, T. and Salerno, C. (2007) Assessing the Cost
Efciency of Italian Universities, Education Economics,
15/4: 455471.
Aksnes, D.W. and Rip, A. (2009) Researchers Perceptions of
Citations, Research Policy, 38: 895905.
Arvanitis, S., Kubli, U. and Woerter, M. (2003)
University-Industry Knowledge and Technology Transfer in
Switzerland: What University Scientists Think about
Co-operation with Private Enterprises, Research Policy,
37/10: 18651883.
Azoulay, P., Ding, W. and Stuart, T. (2006) The Impact of
Academic Patenting on the Rate, Quality and Direction of
(public) Research, The Journal of Industrial Economics, 52/4:
637676.
. (2007) The Determinants of Faculty Patenting Behavior:
Demographics or Opportunities, Journal of Economic
Behavior and Organization, 63/4: 599623.
Banker, R. D., Charnes, A. and Cooper, W. (1984) Models for
Estimation of Technical and Scale Inefciencies in Data
Envelopment Analysis, Management Science, 30: 10781092.
Barker, K. (2007) The UK Research Assessment Exercise: the
Evolution of a National Research Evaluation System,
Research Evaluation, 16/1: 312.
Beath, J., Owen, R.F., Poyago-Theotoky, J. and Ulph, D.
(2003) Optimal Incentives for Income-Generation in
Universities: the Rule of Thumb for the Compton tax,
Internal Journal of Industrial Organization, 21/9: 13011322.
Bonaccorsi, A., Daraio, C. and Simar, L. (2006) Advanced
Indicators of Productivity of Universities. An Application
of Robust Nonparametric Methods to Italian data,
Scientometrics, 66/2: 389410.
Brennan, J. and Teichler, U. (2008) The Future of Higher
Education and of Higher Education Research, Higher
Education, 56: 259264.
Breschi, S., Lissoni, F. and Montobbio, F. (2005) From
Publishing to Patenting: Do Productive Scientists Turn into
Academic Inventors?, Revue dEconomie Industrielle, 110:
75102.
. (2008) University Patenting and Scientic Productivity. A
Quantitative Study of Italian Academic Inventors, European
Management Review, The Journal of the European Academy
of Management, 5/2: 91109.
Buccola, S., Ervin, D. and Yang, H. (2009) Research Choice
and Finance in University Bioscience, Southern Economic
Journal, 75/4: 12381255.
Chang, Y., Chen, M., Hua, M. and Yang, P.Y. (2006)
Managing Academic Innovation in Taiwan: Towards a
Scientic-Economic Framework, Technological Forecasting
& Social Change, 73/2: 199213.
Charnes, A., Cooper, W.W. and Rhodes, E. (1978) Measuring
the Efciency of Decision Making Units, European Journal
of Operational Research, 2: 429444.
12 . T. Agasisti et al.
Coccia, M. and Rolfo, S. (2008) Strategic Change of Public
Research Units in their Scientic Activity, Technovation,
28/8: 485494.
Cooper, W.W., Seiford, L.M. and Tone, K. (2006) Introduction
to Data Envelopment Analysis and its Uses. New York:
Springer.
Czarnitzki, D., Glanzel, W. and Hussinger, K. (2007) Patent
and Publication Activities of German Professors: An
Empirical Assessment of their Co-Activity, Research
Evaluation, 16/4: 311319.
Dai, Y., Popp, D. and Bretschneider, S. (2005) Institutions and
Intellectual Property: The Inuence of Institutional Forces on
University Patenting, Journal of Policy Analysis and
Management, 24/3: 579598.
Daraio, C. and Simar, L. (2007) Advanced Robust and
Nonparametric Methods in Efciency Analysis: Methodology
and Applications. New York: Springer.
Dewett, D. and Denisi, A.S. (2004) Exploring Scholarly
Reputation: Its More than Just Productivity,
Scientometrics, 60/2: 249272.
Etzkowitz, H., Webster, A. and Healy, P. (1998) Capitalizing
Knowledge: New Intersections of Industry and Academia. New
York: State University of New York Press.
Fabrizio, K.R. and Di Minin, A. (2008) Commercializing the
laboratory: Faculty patenting and the open science
environment, Research Policy, 37: 914931.
Gimenez, V.M. and Martinez, J.L. (2006) Cost Efciency in the
University: a Departmental Evaluation Model, Economics of
Education Review, 25: 543553.
Goldfarb, B. (2008) The Effect of Government Contracting on
Academic Research: Does the Source of Funding Affect
Scientic Output?, Research Policy, 37/1: 4158.
Gulbrandsen, M. and Smeby, J.C. (2005) Industry Funding and
University Professors Research Performance, Research
Policy, 34/4: 932950.
Haddow, G. and Genoni, P. (2010) Citation Analysis and Peer
Ranking of Australian Social Science Journals,
Scientometrics, article in Press.
Harnad, S. (2008) Validating Research Performance Metrics
against Peer Rankings, Ethics in Science and Environmental
Politics, 8: 103107.
Hayati, Z. and Ebrahimy, S. (2009) Correlation between
Quality and Quantity in Scientic Production: A Case
Study of Iranian Organizations from 1997 to 2006,
Scientometrics, 80/3: 627638.
Hicks, D. (2009) Evolving Regimes of Multi-University
Research Evaluation, Higher Education, 57: 393404.
Johnes, G. and Johnes, J. (1993) Measuring the Research
Performance of UK Economics Departments: an
Application of Data Envelopment Analysis, Oxford
Economic Papers, 45: 332347.
. (1995) Research Funding and Performance in UK
University Departments of Economics: A Frontier
Analysis, Economics of Education Review, 14/3: 301314.
Johnes, J. (2006) Data Envelopment Analysis and its
Application to the Measurement of Efciency in Higher
Education, Economics of Education Review, 25: 273288.
Koksal, G. and Nalcaci, B. (2006) The Relative Efciency of
Departments at a Turkish Engineering College: a Data
Envelopment Analysis, Higher Education, 51/1: 173289.
Kroll, H. and Schiller, D. (2010) Establishing an Interface between
Public Sector Applied Research and the Chinese Enterprise
Sector: Preparing for 2020, Technovation, 30/2: 117119.
Lissoni, F., Mairesse, J., Montobbio, F. and Pezzoni, M. (2009)
Scientic Productivity and Academic Promotion: A Study on
French and Italian Physicists, Industrial and Corporate
Change, Forthcoming (Working paper n. 27/2009,
Knowledge, Internationalization and Technology Studies
Universita` commerciale Luigi Bocconi, Milan, Italy).
Madden, G., Savage, S. and Kemp, S. (1997) Measuring
Public Sector Efciency: a Study of Economics Departments
at Australian Universities, Education Economics, 5/2:
153167.
Manjarres-Henriquez, L., Gutierrez-Gracia, A., Carrion-Garcia,
A. and Vega-Jurado, J. (2009) The Effects of University-
Industry Relationships and Academic Research On
Scientic Performance: Synergy or Substitution?, Research
in Higher Education, 50/8: 795811.
Meyer, M. (2006) Knowledge Integrators or Weak Links? An
Exploratory Comparison of Patenting Researchers with their
Non-Inventing Peers in Nano-Science and Technology,
Scientometrics, 68/3: 545560.
Moed, H.F. (2007) The Future of Research Evaluation Rests
with an Intelligent Combination of Advanced Metrics and
Transparent Peer Review, Science and Public Policy, 34/8:
575583.
. (2008) UK Research Assessment Exercises: Informed
Judgments on Research Quality or Quantity?,
Scientometrics, 74/1: 153161.
Moreno, A.A. and Tadepali, R. (2002) Assessing Academic
Department Efciency at a Public University, Managerial
and Decision Economics, 23: 385397.
Pesenti, R. and Ukovich, W. (1996) Evaluating Academic
Activities Using DEA, Internal Report, DEEI, Universita`
di Trieste.
Rizzi, D. (1999) Lefcienza dei dipartimenti dellUniversita` Ca`
Foscari di Venezia via DEA e DFA (The efciency of depart-
ments at University Ca` Foscari through DEA and DFA)
Working Paper, Universita` Ca` Foscari di Venezia,
Dipartimento di Scienze Economiche.
Roessner, D. (2000) Quantitative and Qualitative Methods and
Measures in the Evaluation of Research, Research
Evaluation, 8/2: 125132.
Sala, A., Landoni, P. and Verganti, R. (2011) R&D networks:
an evaluation framework, International Journal of Technology
Management, 53/1: 1943.
Stephan, P.E., Gurmu, S., Sumell, A.J. and Black, G. (2007)
Whos Patenting in the University? Evidence from the Survey
of Doctorate Recipients, Economics of Innovation and New
Technology, 16/2: 7199.
Thursby, J.G. (2000) What Do We Say about Ourselves and
What Does It Mean? Yet Another Look at Economics
Department Research, Journal of Economic Literature, 38:
383404.
Van Looy, B., Callaert, J. and Debackere, K. (2006)
Publication and Patent Behavior of Academic Researchers:
Conicting, Reinforcing or Merely Co-Existing?, Research
Policy, 35/3: 596608.
Van Looy, B., Landoni, P., Callaert, J., Van Pottelsberghe, B.,
Sapsalis, E. and Debackere, K. (2011) Entrepreneurial
Effectiveness of European Universities: An Empirical
Assessment of Antecedents and Trade-Offs, Research
Policy, 40: 553564.
Zhu, J. (2003) Quantitative Models for Performance
Evaluation and Benchmarking. Boston, MA: Kluwer
Academic Publisher.
 Evaluating the performance of academic departments . 13

Appendix 1
Table A1. Virtual weights (details by department)

Dep number Dep eld Grants 1 (%) Grants 2 (%) Ext (%) Pub (%) H (%) Cit (%)

2 BIO 0.00 0.00 0.67 0.00 62.27 37.06

3 BIO 7.17 3.44 0.00 4.26 21.76 63.36
20 BIO 37.34 0.00 5.38 57.29 0.00 0.00
31 BIO 88.31 11.69 0.00 0.00 0.00 0.00
32 BIO 0.00 63.60 4.77 0.00 0.00 31.63
42 BIO 0.00 0.00 0.00 100.00 0.00 0.00
43 BIO 7.33 0.00 0.00 0.00 57.57 35.10
44 BIO 0.00 31.73 0.00 0.00 18.16 50.11
47 BIO 0.00 0.00 0.00 68.22 0.00 31.78
48 BIO 0.00 43.68 9.78 0.00 46.54 0.00
54 BIO 20.71 46.43 0.00 0.00 12.28 20.58
61 BIO 24.28 0.00 0.00 67.19 0.00 8.52
64 BIO 0.00 13.10 12.31 0.00 0.00 74.59
65 BIO 25.65 16.15 1.03 0.00 0.00 57.18
67 BIO 16.14 83.86 0.00 0.00 0.00 0.00
70 CHIM 0.00 0.00 1.32 0.12 0.00 98.56
13 CHIM 22.79 0.00 53.29 0.00 9.58 14.34
16 CHIM 0.00 0.00 0.00 100.00 0.00 0.00
37 CHIM 0.00 0.00 0.00 100.00 0.00 0.00
38 CHIM 9.69 2.61 0.00 0.00 50.79 36.92
39 CHIM 0.00 0.00 20.01 79.99 0.00 0.00
46 CHIM 0.00 40.11 0.00 29.19 0.00 30.70
49 CHIM 0.00 18.74 0.00 81.26 0.00 0.00
62 CHIM 0.00 15.18 3.57 0.00 81.25 0.00
66 CHIM 7.21 0.00 3.27 0.00 56.44 33.08
68 FIS 0.00 67.38 0.00 20.92 0.00 11.69
10 FIS 0.00 0.00 0.00 0.00 67.06 32.94
14 FIS 0.00 2.20 55.28 0.00 24.37 18.15
18 FIS 2.56 0.00 12.82 0.00 84.62 0.00
33 FIS 7.36 0.00 0.00 2.31 63.51 26.82
52 FIS 0.00 48.22 51.78 0.00 0.00 0.00
55 FIS 0.00 33.03 0.00 0.00 26.25 40.72
56 FIS 20.89 64.61 0.00 14.51 0.00 0.00
59 ING 13.70 10.59 0.00 26.42 49.29 0.00
4 ING 0.00 0.00 0.00 0.00 100.00 0.00
5 ING 34.12 65.88 0.00 0.00 0.00 0.00
6 ING 0.97 0.00 17.86 35.45 0.00 45.72
7 ING 0.00 15.95 24.98 0.00 6.86 52.20
8 ING 22.96 77.04 0.00 0.00 0.00 0.00
9 ING 0.00 0.00 20.35 79.65 0.00 0.00
11 ING 28.44 0.00 50.09 0.00 21.47 0.00
12 ING 0.00 0.00 0.00 84.72 4.00 11.28
15 ING 0.00 93.80 6.20 0.00 0.00 0.00
19 ING 9.55 0.00 0.52 1.63 13.51 74.80
25 ING 0.00 13.50 2.45 0.00 0.00 84.05
29 ING 36.18 0.00 4.42 0.00 59.40 0.00
35 ING 2.30 0.00 0.00 0.00 63.30 34.40
40 ING 0.00 0.00 6.81 0.00 93.19 0.00
41 ING 0.00 0.00 15.66 0.00 84.34 0.00
50 ING 0.00 0.00 7.63 0.00 35.00 57.37
51 ING 0.00 0.00 8.79 0.00 91.21 0.00
53 MED 7.51 13.41 2.22 0.00 76.87 0.00
1 MED 32.22 4.06 0.00 52.34 11.39 0.00
17 MED 0.00 0.00 0.00 7.39 92.61 0.00
21 MED 36.52 27.57 9.97 25.95 0.00 0.00
22 MED 0.00 0.45 5.67 93.87 0.00 0.00
23 MED 21.99 34.97 0.00 20.32 22.72 0.00
24 MED 0.00 0.00 1.77 64.05 5.43 28.75
26 MED 0.00 0.00 17.52 82.48 0.00 0.00
27 MED 0.00 24.19 19.61 56.21 0.00 0.00
28 MED 0.00 0.00 10.23 89.77 0.00 0.00
30 MED 0.00 90.90 9.10 0.00 0.00 0.00
34 MED 0.00 0.00 0.00 100.00 0.00 0.00
36 MED 9.02 35.55 0.00 5.33 26.51 23.58
45 MED 100.00 0.00 0.00 0.00 0.00 0.00
57 MED 0.00 41.40 0.00 0.00 0.00 58.60
58 MED 0.00 52.25 1.62 0.00 17.38 28.76
60 MED 18.55 16.37 26.67 0.00 9.44 28.97
63 0.00 0.00 9.53 90.47 0.00 0.00
14 . T. Agasisti et al.

Table A2. Inputs and outputs (standardized) slacks

Dep number Inputs Outputs

Lab 2 Staff 2 Grants 1 Grants 2 Ext Pub H Cit

2 0.433 0.000 1.159 0.533 0.000 0.266 0.000 0.000

3 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
20 0.079 0.000 0.000 0.124 0.000 0.000 0.235 0.356
31 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
32 0.000 0.000 2.035 0.000 0.000 0.277 0.005 0.000
42 1.367 0.000 2.709 0.035 0.031 0.000 0.318 0.125
43 0.000 0.000 0.000 0.149 0.306 0.099 0.000 0.000
44 0.629 0.000 1.895 0.000 0.623 1.444 0.000 0.000
47 1.073 0.000 3.267 1.067 0.958 0.000 0.110 0.000
48 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
54 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
61 0.000 0.000 0.000 1.065 0.423 0.000 0.230 0.000
64 0.000 0.107 1.844 0.000 0.000 0.784 0.317 0.000
65 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
67 0.000 0.000 0.000 0.000 0.938 0.066 0.197 0.466
70 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
13 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
16 2.249 0.000 0.083 1.327 0.636 0.000 0.520 0.374
37 0.012 0.000 0.000 0.216 0.200 0.000 0.025 0.013
38 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
39 0.000 0.000 0.844 0.096 0.000 0.000 0.224 0.376
46 0.000 0.000 1.659 0.000 0.391 0.000 0.126 0.003
49 0.159 0.000 1.056 0.000 0.408 0.000 0.013 0.036
62 0.117 0.000 0.040 0.000 0.000 0.095 0.000 0.114
66 0.000 0.000 0.000 0.274 0.000 0.101 0.000 0.000
68 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
10 0.000 0.000 0.743 0.641 0.504 0.177 0.000 0.000
14 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
18 0.274 0.000 0.000 0.137 0.000 0.223 0.000 0.190
33 1.310 0.000 0.000 1.808 1.712 0.000 0.000 0.000
52 0.000 0.000 0.135 0.000 0.000 0.092 0.429 0.875
55 0.000 0.000 0.930 0.000 0.217 0.002 0.000 0.000
56 0.755 0.000 0.000 0.000 0.448 0.000 0.111 0.103
59 0.000 0.000 0.000 0.000 0.307 0.000 0.000 0.151
4 0.597 0.000 2.104 1.129 0.259 0.248 0.000 0.186
5 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
6 0.120 0.000 0.000 0.039 0.000 0.000 0.249 0.000
7 0.628 0.000 2.695 0.000 0.000 1.318 0.000 0.000
8 0.000 0.000 0.000 0.000 0.146 0.017 0.335 0.752
9 0.870 0.000 1.317 0.050 0.000 0.000 0.088 0.436
11 0.567 0.000 0.000 0.829 0.000 0.160 0.000 0.175
12 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
15 0.000 0.000 1.791 0.000 0.000 1.628 1.232 0.976
19 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
25 0.502 0.000 0.409 0.000 0.000 0.139 0.265 0.000
29 0.021 0.000 0.000 0.187 0.000 0.023 0.000 0.738
35 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
40 0.006 0.000 0.875 0.580 0.000 0.093 0.000 0.517
41 0.000 0.000 1.151 0.245 0.000 1.293 0.000 0.150
50 0.919 0.000 0.008 0.038 0.000 0.431 0.000 0.000
51 1.447 0.000 0.008 0.395 0.000 0.318 0.000 0.361
53 0.000 0.000 0.000 0.000 0.000 0.705 0.000 0.025
1 0.493 0.000 0.000 0.000 0.392 0.000 0.000 0.164
17 0.000 0.000 0.362 0.512 0.621 0.000 0.000 0.241
21 0.141 0.000 0.000 0.000 0.000 0.000 0.111 0.060
22 0.000 0.000 0.012 0.000 0.000 0.000 0.395 0.843
23 0.260 0.000 0.000 0.000 0.373 0.000 0.000 0.292
24 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
26 0.778 0.000 0.971 0.123 0.000 0.000 0.476 0.607
27 2.172 0.000 2.931 0.000 0.000 0.000 0.101 0.113
28 0.086 0.000 2.253 0.247 0.000 0.000 0.744 0.744
30 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
34 0.279 0.000 0.842 0.290 0.512 0.000 0.214 0.167
36 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
45 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
57 0.000 2.252 3.092 0.000 1.075 2.263 0.819 0.000
58 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
60 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
63 0.361 0.000 0.683 0.600 0.000 0.000 0.423 0.560