Professional Documents
Culture Documents
How Not To Measure Innovation
How Not To Measure Innovation
=
Y
i
is the country is normalised value, x
i
is the countrys value for the original scale, min(x) and
max(x) are the respective minimum and maximum values which have been attained for this indicator
across all countries. In this way, the normalised indicators Y
i
N
have values lying between 0 (laggard, x
i
=min (x)) and 1 (leader, x
i
=max (x)).
The problem of weighting
The next problem concerns weighting: as mentioned, the SII aggregates 25 indicators into a single
number; in this process, all indicators receive the same weight
ii
w
i
= 1 / 25 = 0.04. This means that all
indicators are equally important which is a rather heroic assumption given the indicators vastly
different coverage (just try to compare the share of the population with tertiary education and venture
capital for start-ups). Certainly, a weighting regime which is beyond all doubt is unattainable; the EIS,
however, takes a very short cut and refrains from any attempt at defining such a weighting regime. This
is justified in the Report on European Innovation Scoreboard 2005 (Sajeva et al., 2005) with the aim to
keep the weighting as simple as possible. To be fair, they did not reach this decision without giv-
ing it some thought: the authors examined a total of 4 different weighting methods:
the budget allocation method, which determines the relative weights of the dimensions and their
indicators on the basis of an expert survey;
equal weighting;
Factor analysis to correct for overlaps between indicators; and
JOANNEUM RESEARCH Institute of Technology and Regional Policy 12
Benefit of the doubt, using weights which are tailored to each country so that this country attains
the best possible ranking.
They conclude that the ranking is quite independent from the weighting method; therefore, they chose
the simplest one. Nevertheless, they drew some criticism (see e.g., Grupp and Mogee (2004) or Schu-
bert (2006)). Using linear programming, Schubert calculated weighting vectors intended to rank each
country as good and as bad as possible. In doing so he showed that it was possible to rank every coun-
try within a very wide range, as the following Figure 2 shows:
Figure 2: Attainable rankings using different weighting regimes
0
2
4
6
8
10
12
14
16
FI SE DE BE AT FR LU UK NL DK I E I T PT ES EL
Source: Schubert (2006)
The top positions are quite unambiguous: independently of the weighting regime, Finland (FI) and
Sweden (SE) are among the top three countries; also, Greeces (EL) position at the bottom seems well
established (although with optimal weighting rank 9 is attainable). On the other hand, Luxembourg
(LU) can take literally any position.
In this example, the weighting vectors were unrestricted, i.e. they could assume any value
iii
. To achieve
more realistic results, Schubert repeated his calculations with restricted weighting vectors, forcing the
individual weights into the range of w
i
= [0.02, 0.06], i.e. between about half and double the value of
equal weighting (w
i
= 1 / 26 = 0.0385). Figure 3 shows the result:
JOANNEUM RESEARCH Institute of Technology and Regional Policy 13
Figure 3: Attainable rankings using different (but restricted) weighting regimes
0
2
4
6
8
10
12
14
16
FI SE DE BE DK NL UK AT FR I E I T LU ES PT EL
Quelle: Schubert (2006)
The result is much clearer now: positions at the top and at the bottom are unambiguous and independ-
ent from the weighting vector. In between, the exact positions are within rather narrow margins -
somewhat manipulable. Altogether, this result seems rather close to the conclusion that the weighting
regime is of secondary importance only vindicating the decision of Sajeva et al. (2006) to keep
the weighting as simple as possible (criticism of which, by the way, was Schuberts (2004) and Grupp
and Mogees (2004) point of departure).
5. THE SII 2007
The following section will show to some extent that the result, i.e. a countrys position in the SII is not
only determined by the selection of indicators but also by the availability of the data used. Additionally
we will analyse the results of SII 2007 published in February 2008 in more detail.
For this the following Table 3 gives first of all an at the glance overview of the official country rank-
ings at the date of publication. Hence, the overall 8
th
rank of Austria in 2005 was based on the data
available in 2005, the 13
th
rank in the following year resulted from the data available in 2006 and so
forth.
JOANNEUM RESEARCH Institute of Technology and Regional Policy 14
Table 3: Country ranking on the basis of the SII
2001 2003 2004 2005 2006 2007
1 S FI J P SE SE SE
2 US SE SE CH CH CH
3 FIN CH FI FI FI FI
4 UK IS US J P DK IL
5 J UK CH DK J P DK
6 DK DK DE US DE J P
7 NL DE DK DE US DE
8 IRL NL I S AT LU UK
9 D IE UK BE UK US
10 FIN FR BE NL I S LU
11 A BE FR UK NL IS
12 B NO NL FR FR IE
13 L AT IE I S AT AT
14 E IT NO LU BE NL
15 IRL LU AT IE IE FR
16 GR ES EE NO NO BE
17 P PT SI IT SI CA
18 EL IT EE CZ EE
19 ES SI EE AU
20 PT HU IT NO
21 LU ES ES CZ
22 BG CY CY SI
23 CZ PT MT IT
24 LT LT LT CY
25 HU CZ HU ES
26 MT BG HR MT
27 SK PL SK LT
28 EL SK PT HU
29 LV EL LV EL
30 CY LV PL PT
31 RO MT EL SK
32 PL RO BG PL
33 TR TR RO HR
34 TR BG
35 LV
36 RO
37 TR
EU15-average; = EU25-average;------- EU27-average
Source: EIS 2001, 2003, 2004, 2005, 2006, 2007
The current SII 2007 exhibits an interesting, because piecewise linear trend (Figure 4): the top posi-
tions show a uniformly falling trend, from Sweden (0.73) to Ireland (0.49). The next 5 positions (from
Austrias 0.48 to Belgiums 0.47) are extremely close (all five of them are separated by less than two of
the top-ranked countries are separated on average); considering measurement errors at the level of the
indicators (and the issue of the correct weighting regime), these countries can be viewed as virtually
on par.
After this group, and a shift of about 0.1 points, the downward trend becomes somewhat stronger, fol-
lowed by a once more rather level section (EU-positions 20-25). Not counting Turkey, Latvia and Ro-
mania come last among the EU27.
JOANNEUM RESEARCH Institute of Technology and Regional Policy 15
Figure 4: SII 2007 - Ranking
1
2
3
4
5
6
7
89
1
0
1
1
1
2
1
3
1
4
1
5
1
6
1
7
1
8
1
9
2
0
2
1
2
2
2
3
2
4
2
5
2
6
2
7
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
S
E
C
H
F
I
I
L
D
K
J
P
D
E
U
K
U
S
L
UI
S
I
E
A
T
N
L
F
R
B
E
E
U
C
A
E
E
A
U
N
O
C
Z
S
I
I
T
C
Y
E
S
M
T
L
T
H
U
E
L
P
T
S
K
P
L
H
R
B
G
L
V
R
O
T
R
S
I
I
2
0
0
7
EU27 countries
other countries
EU average
Quelle: EIS 2006
All this implies that a countrys exact position has to be interpreted with some caution: positions within
the middle (Austria to Belgium) and lagging groups (Hungary to Bulgaria) seem rather exchangeable,
with the exact position depending probably more on luck than ability (all that seems undisputed is
their belonging to the respective group of countries). The positions at the very top and bottom, how-
ever, look quite settled.
The SII over time
Over time, the SII shows in accordance with the evidence of rather subdued intertemporal variability
at the level of the indicators a rather smooth trend (c.f. Figure 5):
JOANNEUM RESEARCH Institute of Technology and Regional Policy 16
Figure 5: SII ranking over time
SII over time: Ranking over time:
2
0
0
3
2
0
0
4
2
0
0
5
2
0
0
6
2
0
0
7
Rank
SE 1 1 1 1 1 0
FI 2 2 2 2 2 0
DK 3 3 3 3 3 0
DE 4 4 4 4 4 0
UK 5 5 5 6 5 1
LU 8 6 6 5 6 3
IE 7 9 7 7 7 2
AT 11 11 11 11 8 3
NL 9 8 9 10 9 2
FR 10 10 10 9 10 1
BE 6 7 8 8 11 5
EE 12 13 12 12 12 1
CZ 14 14 14 14 13 1
SI 15 12 13 13 14 3
IT 13 15 15 15 15 2
CY 17 17 17 17 16 1
ES 16 16 16 16 17 1
MT 18 18 18 18 18 0
LT 21 22 21 19 19 3
HU 20 20 20 21 20 1
EL 19 19 19 22 21 3
PT 24 21 22 20 22 4
SK 22 23 23 23 23 1
PL 23 24 24 24 24 1
BG 25 25 25 25 25 0
LV 26 26 26 26 26 0
RO 27 27 27 27 27 0
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
2
0
0
3
2
0
0
4
2
0
0
5
2
0
0
6
2
0
0
7
S
I
I
SE
FI
DK
DE
UK
LU
IE
AT
NL
FR
BE
EE
CZ
SI
IT
CY
ES
MT
LT
HU
EL
PT
SK
PL
BG
LV
RO
Source: EIS
For most countries, the time series of their SII is quite smooth and typically homogeneous (either fal-
ling or rising over the 5 years). Moreover, SII values exhibit a trend towards the mean: above-
average countries losing and below-average countries gaining over time. For the ranking this implies
that changes in positions are moderate as well. Such changes occur almost exclusively within groups of
countries with similar SII values, for example the group AT/BE/FR/IE/NL which always can be found
at the positions 7-11 (if in slightly varying order). Similar can be said for the group CZ/EE/ES/IT/SI
viewing for positions 12-16.
In this inter-temporal comparison, a very important aspect has to be borne in mind when interpreting
(apparent) changes in the published, up-to-date rankings: according to Figure 5, Austrias position has
barely changed during the last 5 years (the ranking has changed, from 11 to 8, but as we have seen this
rather due to indicator noise than to real advances). This, however, is in blatant contrast with the
public excitement surrounding Austrias gaining the 5
th
rank among the European countries in 2005
(from 10 to 5), only to be followed by a fall back to the 9
th
rank in the subsequent year (see Table 3).
The reason for this (apparent) contradiction can be found in the way the time series of Figure 5 was
calculated: this was done using the current (i.e. 2008) list of indicators and weighting method, as well
as and this is much more important on the basis of now (2008) available data. For any year of the
time series, therefore, their values can depart from the official SII of the selfsame year, if
JOANNEUM RESEARCH Institute of Technology and Regional Policy 17
a) the list of indicator is changed (as happened between 2002 und 2003 as well as 2004 and
2006);
b) the weighting method is changed (the weight of each indicator changes with the number of
dimensions and their respective indicators; again, such changes happened in 2002/03 and
2004/05);
c) the availability of data improves.
This last reason points to the fact that, for example, the official SII2005 (s. Table 3) was calculated
on the basis of data which were available in 2006 (when the SII2005 was published), which almost
certainly included other (older) data than the SII for the year 2005 as calculated in 2008 and presented
in Figure 5.
This last point explains the discrepancy between rank #5 in the official EIS2005 as it is shown in Table
3 and rank #11 as shown in Figure 5: even if neither list of indicators nor weighting method have
changed from 2005 to 2006, what has changed is the available data. Especially the 6 indicators which
are based on the CIS exhibit striking differences: for the EIS2005, data from the CISlight (2002) were
used; now (in 2008), when calculating the SII for the year 2005, data from the CIS4 (2004) can be
used. From CISlight to CIS4, however, almost all variables have lost in value (like indicator 4.4, which
has fallen from 10.6 in CISlight to 5.4 in CIS4). The differences between CIS light and CIS4, however,
are not plausible; rather, they point to problems in comparability, brought about by the fact that due
to quite diverse reasons the CISlight almost certainly overestimated most indicators.
This means, among other things, that Austrias #5 ranking among European countries in the EIS2005
was almost certainly too good and that the current #8 position reflects more adequately Austrias
true position. Compared with the time series of EIS values, this #8 position even represents an im-
provement over earlier positions. In the light of the above discussion, however, this conclusion must be
a qualified one: the SII values of the countries in Austrias peer group are quite similar; it seems safe to
assert that Austria is in the middle group, somewhere between ranks #7 and #11; also, Austria seems
safely ahead of the new members and the Mediterraneans, but also solidly behind the top group of
the northern EU members. In short, Austria is squarely in the medium group; whether on position 7, 9,
or 11 depends more or less on chance. What, now, could Austria do to improve its position? Well, the
possibilities for direct intervention are limited.
6. WHAT CAN POLICY DO?
The logical starting point would be indicator 3.3 innovation expenditure as a share of turnover, which
so far has not been available in Austria (in Austrias SII it is therefore not included). If this indicator is
above average, its inclusion would be good for Austrias ranking on the other hand, a modest value
would be bad. Ironically it could be argued that this indicator has to be afforded a good inspection
before it is measured.
On the level of the other 25 indicators, a wait-and-see approach will not witness much change, as
most of them show similar trends in almost all countries. There are, however, four indicators where
Austria is clearly (i.e., more than 20 %) below average (see Appendix). Two of them do not lend them-
selves to short-term improvement, as they represent the top of the education system, which could only
be changed over longer periods: these are 1.1 New S&E graduates per 1000 population aged 20-29 and
1.2 Population with tertiary education per 100 population aged 25-64. As regards the first indicator, it
probably under-estimates Austrias endowment with engineering brainpower, as it only includes terti-
ary degrees (thus not counting the Austrian speciality of the popular Technical Highschools, which
JOANNEUM RESEARCH Institute of Technology and Regional Policy 18
offer engineering training at the secondary level of education). The peculiarities of the second indicator
have been mentioned already (see section 3).
The worst indicator for Austria by far is 3.4 early-stage venture capital. At the same time, it is proba-
bly the one which could be influence most easily in the short term: in 2005, this amounted to 0.012 %
of GDP in Austria. Raising this to the EU-average of 0.022 % would amount to some 35 Mio. - a not
outrageously large sum (if such a sum would find enough worthy recipients, however, is a different
matter).
Below average are almost all indicators of the dimension output application (4.1 4.5), covering
high tech and new products. Besides problems concerning data quality (2 of the 5 indicators are taken
from the CIS) and ambiguity of definitions (the classification of sectors into low-, medium- and high-
tech is not wholly convincing)
iv
, this dimension clearly shows the dilemma faced by would-be pro-
active politics: what should be done to raise the share of high-tech exports, and in a period of just a few
years? Ironically speaking, the only sure way to achieve this would be to ban all low- and medium tech
exports a patently absurd recipe.
Another good example of this kind is also provided by indicator 4.1 Employment in high-tech services
(% of total workforce) which is somewhat below average in Austria. On the other hand, the share of
employees in the Austrian tourism sector is above average. Although reducing this share to the EU-
average would raise Austrias unemployment rate by some 2 percentage points (which would be bad,
although it would result in an unemployment rate still below the EU average), it would ceteris paribus
raise the share of employment in high-tech services (which would be good for Austrias innovation
ranking). Of course, this example sounds absurd but it serves to highlight the fact that shares or
ratios comprise two terms, the numerator and the denominator. A ratio is low if either the nominator
is low or the denominator is large. A ratio which is deemed too low can be brought about by either
component.
In a nutshell, many indicators reflect structures which have evolved over longer time periods; they
cannot be changed overnight (nor should they be)
v
.
Just two (or three) of the indicators could quickly and directly be influenced by politics: 2.1 public
R&D expenditures and 2.4 Share of enterprises receiving public funding for innovation (plus 3.4 early-
stage venture capital, c.f. above). Both indicators have recorded impressive gains over the last years to
the extent that in the meantime, Austria shows (markedly) above-average values. Of course, further
expansion would be possible although it will increasingly be confronted by the problem of efficient
allocation.
As for indicators of business innovation, these are of course shaped by political conditions. However,
the way from political action to entrepreneurial interpretation is probably rather tortuous; in addition,
when compared with other countries, the Austrian system of incentives in this respect is already quite
generous.
In short: possibilities for (short-term) influence by politics on the variables measured by the EIS are
limited only 2 of the 25 indicators can be influenced directly. Many are the result of long-term devel-
opments, which can be guided but not determined by the political system.
7. SOME FINAL REMARKS
To the extent that the included indicators can unambiguously be measured and compared, the EIS thus
gives a snapshot of a countrys position regarding innovation; as such, it provides valuable informa-
tion to policy makers. Due to a certain lack of such unambiguity, the EIS also reflects country-specific
JOANNEUM RESEARCH Institute of Technology and Regional Policy 19
idiosyncrasies (of its education system, its economic structure, etc.), which can and should not hastily
be labelled good or bad. In the longer term, the EIS is in all probability a feasible and practical tool
to judge on the progress of countries towards innovativeness. What is equally probable, however, is
the futility (maybe bordering on the irresponsible) of using the EIS in the attempt at gauging short-term
developments as we have seen, in the short term the indicators tend to be swamped by measurement
noise. The logical conclusion, therefore, would be for the EU to stop the annual ritual of the innovation
ranking, especially its tendency to provide as up-to-date scores as possible using arguably incom-
patible data for international comparisons as a result (for some countries, some indicators are based on
more recent data than for other countries. The result is an unrealistic variability in the ranking a vari-
ability which almost completely disappears when the EIS is calculated on the basis of older, more com-
patible data).
From a methodological point of view, calculating a composite index is the most critical part of the EIS
exercise. One of the main arguments for the SII is that it is as useful tool for policy making and
consequently much easier to interpret than trying to find a common trend in many separate indicators
(Sajeva et al. 2005, Arundel et al. 2008). However, in our opinion the opposite is true. The use of an
aggregate, single figure is low for practical policy purposes as it is neither immediately transparent nor
does it imply any specific action to be taken. Neither the SII nor the single indicators are related to
typical areas of policy intervention. In general, innovation policy has to take into account market envi-
ronments, technology developments and specific barriers to innovation at different types of enterprises
very carefully (SMEs, start-ups, large companies, export vs. home market orientation etc.) in order to
design an appropriate policy intervention. Thus, innovation policy has to take into account the specific
institutional and economic environment of a country. It would be helpful to combine the publication of
the EIS results with detailed background information on the features of the respective (national) inno-
vation system that may affect the EIS results. As an example for this approach the OECD Economic
Survey for Austria (OECD 2007a) can be mentioned. By using different data sources and by taking
specific policy initiatives into account the OECD Report results into the following policy recommenda-
tions for making innovation policy in Austria more effective. The headlines of these recommendations
are the following:
Simplifying the institutional framework for innovation policy.
Ensure efficiency of innovation subsidies.
Improve product market competition.
Improve conditions for start-ups.
Ease immigration of skilled workers and researchers.
Improve human capital development.
Except the last recommendation no hint for these policy interventions can be found on the basis of the
EIS. A ranking may attract short term attention by the general public at the cost of oversimplifying the
issue and lowering its connection to policy making.
On the other hand, much of the European Commissions powers with respect to national policies rest
with naming and shaming, which, in the case of the reception of the EIS in Austria, for example, has
arguably fulfilled its role (even if the initial uproar has not always been followed by sober and sensible
policies). Whether this approach will work for long, however, is an open question.
JOANNEUM RESEARCH Institute of Technology and Regional Policy 20
8. REFERENCES
Arundel, A., H. Hollanders (2008), Innovation Scoreboards: indicators and policy use; in: Nauwelaers, C.,
R. Wintjes (eds.) (2008), Innovation Policy in Europe, Edward Elgar.
European Commission (2000), Structural indicators; COM(2000) 594 final.
European Commission (2005), European Innovation Scoreboard 2005. Comparative Analysis of Innova-
tion Performance; European Trend Chart on Innovation
Giering, C., Metz, A. (2004), Laboratory for Integration. Opportunities and Risks of the Open Method of
Coordination; Reform Spotlight 2004/02
Goetschy, J. (2003), The open method of coordination and EU integration; Studienbrief 2-010-0204,
Hochschulverbund distance learing.
Grupp, H., M.E. Mogee, (2004), Indicators for national science and technology policy: how robust are
composite indicators?, Research Policy 33, pp 1373-1384
Innometrics (2008), EIS 2007 - Comparative analysis of innovation performance.
OECD (2007a), Economic Survey Austrian, Paris.
Rgent, S. (2002), The Open Method of Co-ordination: A supranational form of governance?; Interna-
tional Institute for Labour Studies, DP/137/2002.
Sajeva, M., D. Gatelli, S. Tarantola, H. Hollanders (2005) Methodology Report on European Innovation
Scoreboard 2005; European Trend Chart on Innovation, European Commission.
Schibany, A., G. Streicher, H. Gassler (2006), sterreich im Kontext des Lissabon- und Barcelonaprozes-
ses; InTeReg Research Report Nr. 52-2006, J oanneum Research.
Schibany, A., G. Streicher, H. Gassler (2007), Der European Innovation Scoreboard: Vom Nutzen und
Nachteil indikatorgeleiteter Lnderrankings; J oanneum Research, InTeReg Reserach Report Nr. 65-
2007.
Schibany, A., H. Gassler, G. Streicher (2007), High Tech or Not Tech; InTeReg Working Paper Nr. 35-
2007, J oanneum Research.
Schubert, T. (2006), How Robust are Rankings of Composite Indicators when Weights are Changing,
Manuskript Fraunhofer ISI.
Veugelers, R. (2007), Developments in EU Statistics on Science, Technology and Innovation: Taking
Stock and Moving Closer to Evvidence-based Policy Analysis, in: OECD (2007), Science, Technology
and Innovation Indicators in a Changing World, Paris.
JOANNEUM RESEARCH Institute of Technology and Regional Policy 21
Appendix
Figure 6: Austrias relative position in the EIS
EI S 2007 I nnovation performance (relative to EU27)
76
77
96
136
110
115
137
96
198
150
85
0
6
98
142
89
68
71
86
102
152
121
145
205
191
INPUT - Innovation Drivers
1.1 New S&E graduates aged 20-29
1.2 Population with tertiary education aged 25-64
1.3 Broadband penetration rate
1.4 Participation in life-long learning
1.5 Youth education attainment level (>upper sec.education)
INPUT - Knowledge creation
2.1 Public R&Dexpenditures
2.2 Business R&Dexpenditures
2.3 Share of medium-high-tech and high-tech R&D
2.4 Share of enterprises receiving public funding for
INPUT - Innovation & entrepreneurship
3.1 SMEs innovating in-house
3.2 Innovative SMEs co-operating with others
3.3 Innovation expenditures
3.4 Early-stage venture capital
3.5 ICT expenditures
3.6 SMEs introduced organisational innovation
OUTPUT - Application
4.1 Employment in high-tech services
4.2 Exports of high technology products
4.3 Sales of new-to-market products
4.4 Sales of new-to-firm products
4.5 Employment in medium-high and high-tech manufacturing
OUTPUT - Intellectual property
5.1 EPO patents per million population
5.2 USPTO patents per million population
5.3 Triad patents per million population
5.4 Community trademarks per million population
5.5 Community industrial designs per million population
Source: EIS 2007
i
See below.
ii
To be more precise, it is the 5 intermediate indicators which receive equal weights: the 25 indicators are
grouped into 5 intermediate indicators (dimensions: innovation driver, knowledge generation, innovation
JOANNEUM RESEARCH Institute of Technology and Regional Policy 22
and entrepreneurship, application, intellectual property). As these dimensions contain 4, 5 or 6 indicators,
there is not an absolutely equal weighting at the level of the 25 indicators.
iii
However, Schuberts results are somewhat unclear: if really ANY weighting was possible, a weight of 1
for some indicator and 0 for every other indicator would be possible as well. Under these circumstances, the
ranking for this composite indicator would be the same as the ranking for the indicator with weight 1.
However, the indicator least favourable for Finland and Sweden is 4.4 Share of turnover with new-to-the-
firm products, where Finland comes 12
th
and Sweden 13
th
and these would at least be the worst positions
that Finland and Sweden could be assigned using this very peculiar weighting vector. Similarly for Greece,
whose second place in 3.3 innovation expenditure should determine the lower bound of its best position in
the composite indicator. The range of attainable positions would therefore be even larger than presented by
Schubert. Maybe the weighting vector was not as free as Schubert reports (requiring each indicator to
receive at least a small positive weight). Whatever the reason, Schubert shows impressively that with clever
weighting (and a bit of goodwill) quite diverse results can be achieved.
iv
For example, the OECD classification does not allow for high-tech niches in medium-tech sectors like
the manufacture of machinery, which is quite important in Austria. On the other hand, mere assemblage of
hardware modules is included in the high-tech sector manufacture of computers.
v
The oft-cited example of Finland, which experienced a well-nigh revolutionary structural change within
just a couple of years, is one that should not be recommended for emulation frivolously: it was the result of a
deep recession following the collapse of the Soviet Union, and with it the markets of Finlands low-tech
exports. Moreover, Finnish high-tech is to a large degree represented by a single enterprise not a text-book
example of a balanced portfolio.