INTOSAI Working Group on Key National Indicators

WHITE PAPER
ON KEY NATIONAL INDICATORS
 2

Content

Introduction............................................................................................................................... 3
Problem overview ..................................................................................................................... 9
Countries’ experiences in the development and use of KNI ............................................. 9
International organizations’ experiences in progress measurement ............................... 10
SAIs’ experiences in the use of KNI................................................................................ 12
Principles and Guidelines ....................................................................................................... 13
Principles for SAIs’ application of KNI............................................................................. 13
The role of SAIs in the development, assessment, and use of KNI systems .................. 14
KNI: Guide to terms and concepts .................................................................................. 15
Q: What is measured? .............................................................................................. 15
Q: What are KNI? ..................................................................................................... 15
Q: What are KNI systems? ....................................................................................... 16
Q: How is data quality defined? ................................................................................ 16
KNI in sustainable development monitoring.................................................................... 16
Guidelines for knowledge-based economies .................................................................. 17
Guidelines for the development and use of KNI in developing economies (the example
of the Commonwealth of Independent States member-states) ....................................... 19
Final Statement ...................................................................................................................... 20
Annex A .................................................................................................................................. 22
Annex B .................................................................................................................................. 28
 3

Introduction

As the global financial crisis spurs critical thought about socio-economic development
strategies around the world, the discussion of developing and using Key National Indicators
(KNI) is particularly timely. This white paper focuses primarily on KNI as a necessary tool for
the effective evaluation of national strategies, and the role of supreme audit institutions
(SAIs) in their development and use. We hope that in the future, this paper can become a
key document for understanding the specific ways in which SAIs can help achieve national
goals through effective audit methods.
The cross-links developed between this white paper and INTOSAI’s Knowledge Base on KNI
are of particular significance. Consolidating information on the theory and practice of KNI use
will not only provide professionals with the tools necessary to prepare and conduct audits
and use the results, but will also help to create a common information space for all matters
relating to SAIs and the development and use of KNI.
Universal and specific recommendations are presented in this white paper. Universal
recommendations include principles for SAIs’ application of KNI and a guide to KNI terms
and concepts that provides definitions and various examples of real-world applications. In
addition, recommendations on the use of Key National Indicators in sustainable development
monitoring are provided. Specific recommendations on the use of KNI in SAIs’ activity relate
to using KNI to describe the processes of knowledge-based economies and societies and
include recommendations to the nations that are already moving in this direction (for
example, members of the Commonwealth of Independent States).
This white paper is currently descriptive, identifying common methodological approaches
related to using KNI in auditing. Developing recommendations that are applicable to SAIs
with different authorities or in countries with different levels of socio-economic development
will require further work. It is essential that SAIs in countries with and without KNI systems
participate in the future development of this document. Thus, this document should not be
considered as static, as it will be continually updated to serve as an effective tool in the
development and use of KNI.
The White Paper on KNI is developed by the members of the INTOSAI Working Group on
KNI. During the preparation of the document the following documents were used:
 Lima Declaration of Guidelines on Auditing Precepts adopted in 1977 at the IX
Congress of INTOSAI;
 Decision of the XIX INTOSAI Congress on the establishment of the Working Group
on KNI within the framework of Strategic Goal 3;
 4

 Terms of Reference of the INTOSAI Working Group on KNI; and

 Materials, prepared within the framework of the realization of INTOSAI Working
Group on KNI subprojects, including:
o review of countries experience in the development and use of Key National
Indicators;
o overview of international organizations experience in progress measurement;
o principles for SAIs’ application of Key National Indicators;
o Key National Indicators: Guide to terms and concepts;
o the role of Key National Indicators in sustainable development monitoring;
o overview of a framework for Key National Indicators describing the processes
of knowledge-based economy and society; and
o guidelines for the use of Key National Indicators in performance audits within
the framework of the Commonwealth of Independent States (CIS).
The outcomes of this work should be of interest not only to Working Group members but to
all INTOSAI members, and it is essential that SAIs in countries with KNI systems or sets as
well as those where KNI have not yet been developed participate in the development of this
document.
The main goal of the most countries’ long-term policy is to support safe and sustainable
development. In order to achieve the desired result, namely, to improve the quality of life and
promote the effective use of national resources, it is necessary to realize the importance of
issues related to strategic management and the long-term development strategies and
programs. In order to achieve strategic goals and priorities concerning worldwide, regional
and national development there is an obvious need to develop KNI that will allow audit the
realization of socio-economic development strategies and their compatibility with global
development goals.
This is a new mission of SAIs arising from the modern challenges, which doesn’t merely
involve the exchange of best national practices and joint expert and analytical work, but also
the participation of all interested parties. The format of the White Paper allows the
implementation of such an approach, because it implies not only a set of proposals and
recommendations of interested individuals and entities on a specific topic, ways, methods
and tools for their application in practice, but also is a form of public statement of intent of
relevant institutions, and statements involving public support.
The White Paper on KNI has the following goals:
 Highlight the importance of the development and use of KNI in assessment systems
of socio-economic development;
 Support a comprehensive approach to the development and use of KNI;
 Enhance SAIs’ role in the assessment of effectiveness and efficiency of government
activities on the basis of KNI;
 Support the international role of INTOSAI in promoting the development and use of
KNI;
 Build the basis for strengthening cooperation in the sphere of progress measurement
between INTOSAI and other international organizations engaged in such researches;
 Promote the exchange of best practices in the development and use of KNI and
dissemination of experience in the countries that lacking of the KNI system;
 5

 Assist countries and organizations interested in the development and use of KNI in
the sphere of policy and decision making processes; and
 Promote the continuous monitoring of the countries strategic goals compliance.
The White Paper on KNI has the following structure:
 Introduction,
 Executive summary,
 Problem overview,
 Principles and Guidelines,
 Final statement, and
 Annexes.
This paper is intended largely for SAIs and is aimed at creating common approaches,
methodologies, and standards for the application of KNI during the evaluation of economy,
efficiency, and effectiveness of development strategies. Using the conclusions given in this
paper, each SAI can develop a detailed program of actions for the assessment of
development models’ effectiveness, and ways to achieve stated goals, whether or not the
SAI’s country already has a complete KNI system.
 6

Executive summary

I. The White Paper on KNI contains the results of the proceedings of the INTOSAI
Working Group on KNI, established according to the recommendation of the XIX
INCOSAI, held in Mexico City in 2007.
II. There is no single definition of KNI. In this paper, we define KNI as a specific set of
indicators that measure economic and social progress in achieving national
goals in the respective areas. The system of KNI is defined as an organized effort
to assemble and disseminate a group of indicators that together tell a story
about the position and progress of a nation.
III. The main goals of the Working Group were to:
– promote the use of KNI as a tool of SAIs’ audit activities,
– formulate the general rules (guidelines) of SAI use of KNI,
– facilitate the cooperation between SAIs using or intending to use KNI in their
work, and
– disseminate the examples of good practices for developing and using KNI.

IV. In countries using KNI, SAIs may be involved in promoting the development,
selection, use, and continuous improvement of KNI. The role of SAIs in this process
largely depends on political, legislative, and administrative systems of each country
and each SAI’s mandate.
V. The review of various countries’ experiences in using KNIs reflects the diversity of
approaches to KNI development and application. Factors such as the type of
economy, the available socio-economic development strategies, the activity of civil
institutions, national traditions, and international obligations all affect the selection of
key indicators.
VI. SAIs can use KNI as an audit tool for an independent evaluation of the effectiveness
of authorities’ decisions. KNI can be also used for the risk analysis in the process of
preparing annual audit plans.
VII. The Working Group proposes a set of rules as the principles for SAIs’ application of
KNI. The proposed principles are generic, i.e., acceptable regardless of levels of
socio-economic development and models of progress. The rules are as follows:
Conditions
1. SAIs’ use of KNI has to be within their mandates and should respect their
independence.
1.1. Direct participation in the construction and improvement of KNI is not in
accordance with SAIs’ prerequisite of independence, however, SAIs
may participate in improving KNI by giving advice.
1.1.2. SAIs’ giving advice on the development and improvement of
KNI have to respect the principles of objectivity and impartiality
and not compromise the principles of independence.
1.1.3. SAIs should ensure that giving advice on development and
improvement of KNI does not lead to conflicts of interest and
does not include management responsibilities or powers.
 7

2. An SAI’s staff must have professional knowledge and experience within the
fields of both the policy area and the methodological questions concerning
KNI.
SAI duties
3. An SAI has to emphasize accountability when evaluating and using KNI in the
audit.
3.1 An SAI has to draw attention to the value of disclosure and
transparency of all aspects of KNI.
3.2. An SAI has to promote the use of KNI in all stages of the budgetary
process, including programming and planning.
Function
4. A KNI system is an instrument for analyzing the implication of public policies,
particularly in implementing performance audits.
4.1 SAI audits of KNI should enable corrective action in the relevant policy
area.
5. An SAI should be able to determine whether a government’s implementation of
KNI is adequate.
5.1 As part of this task, an SAI must evaluate the validity, reliability,
conciseness, completeness, independence, and comparability of a
government’s KNI, and the information systems providing data to the KNI.
Requirements
6. An SAI must evaluate the disclosure of KNI methodologies to assure
transparency in KNI use.
7. When working with KNI, an SAI has to use generally accepted and modern
scientific methods within disciplines such as economics, statistics, social
science, and management science.
Methods
8. When an SAI is using KNI to analyze the implications of public policies, the
selected KNI have to be material in relation to the issue.
8.1 An SAI must evaluate the set of KNI established to illustrate the
progress of the approved policy.
8.2 An SAI must evaluate critically the capability of the stipulated KNI
system in order to increase the number of international comparisons.
9. When evaluating existing KNI, an SAI has to evaluate the risk associated with
not measuring the right issue.
Communication
10. An SAI should evaluate that a government’s communication regarding KNI is
carried out in compliance with general principles of public statistical
information.
10.1 When an audit of KNI reveals weaknesses, an SAI has to present its
findings in such a way that creates opportunities to improve the KNI
system.
VIII. Although SAIs have encouraged the development of KNI systems, they have
generally avoided involvement in the selection of indicators in order to retain their
 8

independence and any possible loss of credibility if the indicators are viewed as
inaccurate or inappropriate.
IX. Within its mandate, SAIs should audit such problems of KNI systems as:
 credibility of information systems using to calculate the values of the KNI, and
 adequacy of the set of KNI to the goals of the multiannual programs and
strategies.
X. An audit mandate is the regulation of the extent to which an SAI can audit public
policies, programs, and organizations. To develop an audit of a knowledge-based
economy within an SAI’s audit mandate, an SAI can undertake these three
evaluations, in succession:
 the evaluation of research and development (R&D) programs;
 the evaluation of progress in the knowledge economy; and
 the evaluation of progress in the knowledge society.
XI. KNI systems can include specially developed indicators that cover all areas of
government activity, as well as a number of traditional macroeconomic indicators
developed by national statistical services. In this regard, SAIs should cooperate
closely with national statistics agencies in their countries concerning the use of
relevant data and review of KNI accuracy and reliability.
 9

Problem overview

Countries’ experiences in the development and use of KNI

KNI today are usually considered as a specific set of indicators that measure economic and
social progress in achieving national goals. Ideally, a KNI system is an element of an overall
strategic management plan. However, the interpretation of the term "Key National Indicators"
varies depending on the country and its system of performance measurement. Currently,
there is quite a varied experience of KNI application. In some countries, KNI are a part of the
strategic planning process and refer to government activity, while in others, the KNI system is
based on traditional macroeconomic indicators, which are developed by national statistical
services.
Models of management development and performance measurement methods largely
depend on a country's existing political, legal, and administrative systems. In some countries,
these processes are centralized, in others, decentralized. The lack of systems of strategic
management and performance measurement at the national level usually means a lack of
audit and monitoring of government socio-economic development strategies. Economic,
social, and environmental indicators in this case are used for current monitoring of socio-
economic development of the state, but not as an element of strategic management.
In many countries, the existence of both a national socio-economic development strategy
and an integrated assessment system of the state of the economy and society is assumed.
In this context, KNI would reflect the highest public priorities and obligations of the state,
helping to enable changes that improve the economy and society while preserving national
identity, sovereignty, and unity. It is important to emphasize that such indicators are topical
when there is a perceived need for integrated development management and that the
processes of the implementation of national socio-economic development strategies and the
development of indicators are interrelated.
In most countries, the development of national indicators is primarily the responsibility of the
government sector, however, in some countries, dialogue between the citizens and decision-
makers forms the basis for the development of national indicators.
The fact that KNI are generally understood in a system rather than individually implies that
the goals, objectives, and development indicators are to be interconnected and
interdependent. However, because this usually depends on the quality of management,
these conditions are not always followed.
To date, the Working Group has identified the following 16 countries as having developed KNI
systems:
 10

United Kingdom Portugal Japan Kiribati

Slovakia Malaysia Albania Indonesia
Mexico Switzerland Poland South Africa
Netherlands Norway Saudi Arabia Greece

The number of KNI can vary greatly depending on the country. The 16 countries on this list,
with a few exceptions, have national development strategy documents, but this does not
necessarily mean that the KNI systems are developed in accordance with them. In most
countries, KNI systems have been formed recently and have existed for less than ten years.
Nations’ international obligations, as well as the indicators they must use to report on their
achievements to international organizations, play an important role in the development of KNI
systems. Moreover, there are examples where international obligations are clearly reflected
in national development strategies.
Some countries without KNI systems have established special institutions to serve some of
the same functions. For example, France established the Commission on the Measurement
of Economic Performance and Social Progress in 2009. In the report prepared by the
members of that commission, which includes Nobel laureates Joseph Stiglitz (2001) and
Amartya Sen (1998), KNI are considered not just as statistical data, but as indicators that
reflect the level of the public consent on the development targets and priorities. Thereby, KNI
based not only on economic but on social aspects are intended to contribute to effective
change management and to the growth of the social welfare and competitiveness of states.
Thus, a review of various countries’ experiences in using KNIs reflects the diversity of
approaches to KNI development and application. Factors such as the type of economy, the
available socio-economic development strategies, the activity of civil institutions, national
traditions, and international obligations all affect the selection of key indicators.

International organizations’ experiences in progress measurement

Numerous international organizations develop and publish sets of indicators that are similar
to or fit the definition of KNI. These sets of indicators can vary substantially in their numbers,
subject scope, frequency of publication, and most importantly, general concepts
(“philosophies”) behind the composition of the set.
Indicators published by international organizations do not directly represent the operations of
the given organization, but rather describe the “state of the world” within the area of that
organization’s interests. Their prime purpose is, therefore, to establish a base for making
international comparisons and evaluating the dynamics of change taking place in certain
countries. Such indicators, when published on a regular basis and when the methodology of
their compilation and ensuring their comparability across countries is accepted as
trustworthy, become important instruments in shaping the perception of individual countries.
They also serve to increase pressure for addressing the problems detected through the
presented data.
Most often the sets of indicators published by international organizations take the shape of
cross-section (by years and countries) tables, presenting selected statistical data from the
range of interests of the given organization. This is the character of indicator sets published,
for instance, by:
 11

 Organization for Economic Cooperation and Development (OECD)1

 Eurostat (Community Statistical Authority of the European Union)2
 International Monetary Fund3
 Food and Agriculture Organization4
 International Labor Organization5
 World Health Organization6
These data are fundamentally significant because they present information in accordance
with uniform methodological rules and are comparable both over time and between different
countries. Still, the number of presented indicators is, as a rule, so extensive that their
analysis fails to provide a synthetic view of the situation in given countries. The fact that such
data carry the imprimatur of international organizations implies that they are accepted as
trustworthy.
The sets of indicators (databases) referred to earlier are generally composed of several
hundred or even several thousand of time series. In order to facilitate access to the most
meaningful data, in several of the databases there is a subset of key indicators, akin to KNI
in character. Examples of these subsets include, for instance, the set of Main Economic
Indicators in the OECD database, or the set of sustainable growth measurements in the
Eurostat database.
More interesting would seem to be the indicators of another type, namely, indicators
designed to appraise the degree of progress in implementing global strategies pursued or
promoted by international organizations. The best examples of such indicators are those sets
measuring progress in meeting the UN’s Millennium Development Goals,7 and those
measuring the implementation of the EU’s “Europe 2020” strategy, which was developed as
a continuation of the Lisbon Strategy.8
Although these kinds of indicator sets occur less frequently, the indicators are still carefully
chosen and strictly bound with strategy goals. This means that while they do not provide a
very comprehensive picture of the nations described by the published indicators, they are
closely focused on crucial issues. This trait brings strategy implementation indicators closer
to the “classic” KNI sets.
The universal character of strategy implementation indicators (the same indicators are used
for assessing the situation in different countries) obviously also has significant drawbacks: a
set of global indicators can, after all, overlook problems which are unique to a given country
but are of fundamental importance. One also notes that the strategy implementation
indicators (both those cited above as examples, and many others as well) are often focused
primarily on assessing the situation in countries with lower levels of socio-economic
development, and therefore may be less useful in countries that are more advanced in this
respect.

1
See Main Economic Indicators, http://stats.oecd.org/index.aspx
2
http://ec.europa.eu/eurostat/data/browse-statistics-by-theme
3
See IMF Data and Statistics, http://www.imf.org/external/data.htm#data
4
http://faostat3.fao.org/home/E
5
http://www.ilo.org/ilostat/faces/help_home/data_by_subject?_adf.ctrl-
state=i6ni3rk0f_304&_afrLoop=236221586977832#!
6
http://www.who.int/gho/themes/en/
7
http://mdgs.un.org/unsd/mdg/Default.aspx
8
http://ec.europa.eu/eurostat/web/europe-2020-indicators/europe-2020-strategy
 12

Some of the indicators in these databases are presented as single, synthetic indicators that
reflect the situation in selected areas. Some of the most prominent indicators of this type
include:

 Human Development Index, a synthetic index of human resources quality, published

annually by the United Nations Development Program9
 Corruption Perception Index, published annually by Transparency International10
 Doing Business Index, which uses a single, synthetic indicator measuring the ease of
starting and carrying out business operations in given countries, published by the
World Bank11
 Global Competitiveness Index and Business Competitiveness Index, indexes
measuring the level of countries’ development, published by the World Economic
Forum12
The reason behind the popularity of these indexes is the fact that compressing numerous
aspects of the analyzed issues to a single, synthetic indicator allows for drawing up a ranking
list, very clearly showing the standing of individual countries. Although the methodology of
these indicators raises questions as to their soundness, they are very significant to how the
evaluated countries are perceived.

SAIs’ experiences in the use of KNI

In countries using KNI, SAIs may be involved in promoting the development, selection, use,
and continuous improvement of KNI. Even so, SAIs must maintain their independence in
order to subsequently use KNI for an independent evaluation of the effectiveness of
authorities’ decisions. The role of SAIs in this process largely depends on political,
legislative, and administrative systems of each country, and the SAI’s mandate.
Integrated performance evaluation systems operate in countries where KNI are already
developed and used. They cover all levels of governance: national, subnational,
governmental sectors, institutions and budget programs. Such performance evaluation
systems imply that goals, objectives and indicators should be consistent and comparable;
however, it usually depends on the quality of governance and a range of planning
procedures.

9
http://hdr.undp.org/en/statistics/
10
http://www.transparency.org/research/cpi/overview
11
http://www.doingbusiness.org/economyrankings/
12
http://www.weforum.org/
 13

Principles and Guidelines

Principles for SAIs’ application of KNI

Since every country in the process of development faces various socio-economic problems,
each country’s development goals require individually-designed KNI. In addition, in the
process of KNI development, it is necessary to take into account not only development
strategies, but also the possible risks of their implementation.
The proposed principles are generic, i.e., acceptable regardless of levels of socio-economic
development and models of progress. Being nonspecific and general is important because
each individual country, depending on political priorities, may have different socio-economic
problems and corresponding lists of KNI.
KNI can be considered as performance audit criteria by which socio-economic development
strategy outcomes, government activity, socio-economic processes, and society condition as
a whole are evaluated.
Conditions
1. SAIs’ use of KNI has to be within their mandates and should respect their
independence.
1.1. Direct participation in the construction and improvement of KNI is not in
accordance with SAIs’ prerequisite of independence, however, SAIs
may participate in improving KNI by giving advice.
1.1.2. SAIs’ giving advice on the development and improvement of
KNI has to respect the principles of objectivity and impartiality
and not compromise the principles of independence.
1.1.3. SAIs should ensure that giving advice on development and
improvement of KNI does not lead to conflicts of interest and
does not include management responsibilities or powers.
2. An SAI’s staff must have professional knowledge and experience within the
fields of both the policy area and the methodological questions concerning
KNI.
SAI duties
3. An SAI has to emphasize accountability when evaluating and using KNI in the
audit.
3.1 An SAI has to draw attention to the value of disclosure and
transparency of all aspects of KNI.
 14

3.2. An SAI has to promote the use of KNI in all stages of the budgetary
process, including programming and planning.
Function
4. A KNI system is an instrument for analyzing the implication of public policies,
particularly in implementing performance audits.
4.1 SAI audits of KNI should enable corrective action in the relevant policy
area.
5. An SAI should be able to determine whether a government’s implementation of
KNI is adequate. .
5.1 As part of this task, an SAI must evaluate validity, reliability, conciseness,
completeness, independence, and comparability of a government’s KNI,
and the information systems providing data to the KNI.
Requirements
6. An SAI must evaluate the disclosure of KNI methodologies to assure
transparency in KNI use.
7. When working with KNI, an SAI has to use generally accepted and modern
scientific methods within disciplines such as economics, statistics, social
science, and management science.
Methods
8. When an SAI is using KNI to analyze the implication of public policies, the
selected KNI have to be material in relation to the issue.
8.1 An SAI must evaluate the set of KNI established to illustrate the
progress of the approved policy.
8.2 An SAI must evaluate critically the capability of the stipulated KNI
system in order to increase the number of international comparisons.
9. When evaluating existing KNI, an SAI has to evaluate the extent to which
there is a risk for not measuring the right issue.
Communication
10. An SAI should evaluate that a government’s communication regarding KNI is
carried out in compliance with general principles of public statistical
information.
10.1 When an audit of KNI reveals weaknesses, an SAI has to present its
findings in such a way that creates opportunities to improve the KNI
system.

The role of SAIs in the development, assessment, and use of KNI systems

Although SAIs have encouraged the development of KNI systems, they have generally
avoided involvement in the selection of indicators in order to retain their independence and
any possible loss of credibility if the indicators are viewed as inaccurate or inappropriate. To
guard against these risks, SAIs can take a number of steps, including limiting their
involvement in design to technical assistance and performing an auditing role after the
indicators are developed.
 15

Beyond development, SAIs can play a number of roles in supporting and using KNI systems
in audit work. SAIs have played a role in assessing the reliability and relevance of KNI and
have used KNI as a basis for assessing government performance. The following questions
may serve as a general guide for SAIs to consider as matters for audit:
 Is there a KNI system in place?
 Is the KNI system linked with the budget development process?
 Are KNI compatible with macroeconomic indicators?
 Are KNI used to report on progress towards international goals?
 Are different indicators used at the national and sub-national levels?
 Are KNI linked with other government indicators and are they harmonized?
 Are there systems in place to monitor achievement of government policies?
 How do national indicators relate to goals or objectives established in legislation?
 Are national indicators valid and reliable measures of national goals? Do they reflect
objectives of legislation? Are there well-established relationships between national
goals and the indicators related to them?
 To what extent are strategies of socio-economic development and government
programs contributing to national goals, as measured by KNI?

KNI: Guide to terms and concepts

The diversity of interpretations of such terms as “progress,” ”Key National Indicators,” “data
quality,” and others makes it necessary for SAIs to formulate a common understanding of
key terms used by SAIs. In this case, not only definitions and terms, but their interpretations
and description of the most correct way to use them, are important. In accordance with this
objective, this guide answers the following questions:
 What is measured?
 What are KNI?
 What are KNI systems?
 How is data quality defined?

Q: What is measured?

A: Progress. In simple terms, progress means that life is getting better for a society as
defined by members of that society. Progress may also be defined as success in attaining or
nearing the goals that are established through a political process or other type of civic
engagement. Progress is multi-dimensional and typically includes economic, social, and
environmental factors along with other areas that people see as important to life (for
example, culture, national security or the quality of governance). Although progress implies
change for the better, any assessment of progress must also include assessment of regress.

Q: What are KNI?

A: Sometimes referred to as “headline indicators,” KNI define a core set of information about
the progress and position of a nation, selected from a range of possibilities. There is no
“right” number of indicators; how the balance is struck between simplicity and breadth of
coverage can vary widely. However, KNI are generally limited to what society considers the
“vital few.” While a set of indicators can include anywhere from a few to dozens, any KNI set
is not intended to be exhaustive, but rather, to provide a summary picture of those conditions
considered to be most important for the progress of a nation. As is the case in defining
 16

progress, the process of selecting KNI is inherently political, representing the aspirations and
values of society.

Q: What are KNI systems?

A: A KNI system, which may be referred to as a “suite of indicators,” is an organized effort to

assemble and disseminate a group of indicators that together tell a story about the position
and progress of a nation. Indicator systems collect information from data collectors and
package it into products and services for leaders, researchers, planners, and citizens, among
others. A KNI system generally includes social, economic and environmental indicators of a
nation to provide an overall picture of the country’s progress and well-being. While many
countries have indicators in one or another of these areas, a KNI system can provide a
comprehensive and balanced view, to help to ensure that one dimension of progress is not
advancing at the expense of another.

Q: How is data quality defined?

A: Data quality can be defined as “fitness for use,” a concept that includes a number of
attributes that contribute to the usefulness of the data from the perspective of the users, such
as relevance, accuracy, credibility, timeliness, accessibility, interpretability, and coherence.
Data quality is ensured by verifying and validating data so as to avoid data limitations, i.e.,
problems with the data sources or the data that may be identified in program evaluations,
independent audits, information systems analyses, etc.

KNI in sustainable development monitoring

In many countries, the concept of sustainable development has become an integral part of
policies, strategies, and programs at regional, national, and local levels. This has
necessitated the urgent development and application of new audit methods and tools.
The concept of sustainable development is best understood as welfare (economic,
environmental, and social) in the short term and the long term. Considering these ideas
together can help ensure that policy making promotes development in the present that does
not compromise the capability of future generations to meet their needs.
To evaluate whether development is sustainable, SAIs should conduct the following types of
audits:
 an audit of targets, to see if they are realistic and are based on proper
understanding and evidence about what needs to be done;
 an audit of indicators, to see if they are relevant and reliable; or
 an audit of the progress revealed by comparing indicators with their associated
targets.
After reviewing the relevance and reliability of targets and indicators, SAIs could develop
suitable criteria. The targets, as commitments, might be taken from national plans and
programs or international treaties adopted. According to the OECD guidance on sustainable
development indicators, for the audit of indicators SAIs might investigate whether indicators:
 have policy relevance, which means that they must:
– show trends over time;
– respond to changes in driving forces, and
 17

– have threshold or reference values against which progress can be measured.

 are analytically sound, for example based on a clear understanding of the goal of
sustainable development;
 are measurable, that is, no matter how attractive the theoretical construction, if an
indicator cannot be measured at reasonable cost, it is not useful.
In connection with the current global financial crisis management, special attention needs to
be given to the indicators describing the efficient utilization of public funds dedicated to crisis
management, the significant growth in public debt service, and public spending.
Furthermore, the present financial crisis also needs the establishment of an international
coordinated system of ‘early warning’, in this connection a closer cooperation between the
SAIs and the international financial organisations, in order to facilitate policy relevance
assessment of the indicators for the SAIs. In that regard, the INTOSAI Governing Board
decided in 2012 to convert the Global Financial Crisis Task Force into a permanent working
group on financial modernization focusing on regulatory reform of financial markets and
institutions. This decision was a clear demonstration of how, given that governance issues
are crosscutting and global, SAIs must work in a crosscutting and global way to be effective
and return value in their respective national contexts.
The many-sided needs for indicators seem to justify the necessity of comparisons of
countries socio-economic development with regards to the experience of mutual monitoring
of the sustainable development of the G-20 countries. The guiding principles of the work
appear to be as follows:
 The mutual effects or services of three dimensions of sustainable development
(economic, environmental, and social) should adequately be balanced by the scope
of indicators.
 Maintaining balance between short and long-term information needs.
 The indicators have to illustrate realistically the trade-offs between the aims and the
actual performances of the three dimensions.
 In the light of great diversity in the sets of indicators for monitoring sustainable
development, it is important to find a good balance between the reduced sets of
‘core’ or ‘headline’ indicators and the very detailed ones.
 To bring the three dimensions of sustainable development together simultaneously
into accounting frameworks that are not in use at present.
 Because of the limitations of some major indicators (e.g.: GDP, productivity) there is
a need for the development and use of alternative indicators, in order to promote
more reliable analyses.

Guidelines for knowledge-based economies

Nowadays, many countries operate within knowledge-based economies and societies (KES).
This is manifested through knowledge-based industries and services as well as institutions in
the economic and social structure, in addition to the growing government’s programming and
funding activities for KES progress. These kinds of changes need to be adequately reflected
in performance auditing, particularly from the accountability perspective. Beyond this,
attention should also be given to the development and understanding (possibilities, causes,
preconditions) perspectives in the case of research and development (R&D) programs, in
particular.
 18

An audit mandate is the regulation of the extent to which an SAI can audit public policies,
programs, and organizations. To develop an audit of a knowledge-based economy within an
SAI’s audit mandate, an SAI can undertake these three evaluations, in succession:
 the evaluation of research and development (R&D) programs;
 the evaluation of progress in the knowledge economy; and
 the evaluation of progress in the knowledge society.
The purposes of performance auditing should be decided on by the SAI for achieving the
following main goals:
 the evaluation of economy, efficiency, and effectiveness of the government’s
activities in support of KES progress;
 the determination of the most important indicators of effectiveness and their sources
of data; and
 the assessment and improvement of political decisions and goals established for
promoting KES progress.
One of the problems hindering the implementation of these tasks is that the information
needs of decision makers are not systematically met or analyzed. Another basic problem
area in performance auditing is the lack of explicit intervention logic and the presence of
poorly defined program objectives, which lead to a very delicate basis for audit criteria.
In order to overcome these inherent difficulties and establish the basis for assessing long-
term results, it is necessary to include explicit intervention methods in future program or fund
designs. Of course, it is also necessary to improve regulation policy to simplify the regulatory
environment and make it more effective and easier to understand.
In addition, auditors should recognize that certain types of KES analyses require a long-term
perspective (e.g., the evaluation of outcomes and socio-economic impacts) and that some
aspects are related to a specific programming period of short or medium terms (e.g.,
program objectives, even within a given scientific field), whereas others are not.
Due to the usual lack of reliable independent information, a special regulation is also needed
that specifies various information sources other than the auditees.
Related directly to the process of the development of KNI, in addition to indicators that are
available and used in a country, SAIs could select indicators from among the knowledge
assessment methodology of the World Bank, and the European innovative scoreboard for
performance auditing of the knowledge economy. In the case of indicators describing the
information society, the community statistics on the information society offers itself as a
useful reservoir for selection. Beyond the benefits of the use of best practices, this way of
enriching the set of domestic indicators makes the indispensable international comparisons
easier.
In general, the range of indicators needed for covering all phases of the development of the
KES starting from input factors up to utilization of outputs and their final economic and social
impacts. Assessing the relationships of outputs to impacts, i.e., the benefit for individuals,
communities, and a given economy and society, poses analytic and methodological
challenges for auditors.
As the result of performance audit in knowledge-based economies, it’s possible to highlight
the following:
 on the basis of principles for SAIs’ application of KNI, SAIs should contribute to the
further development of their respective national systems of indicators by describing
the processes of knowledge economy and information society, as well as ;
 19

 in a ‘knowledge-based’ environment, performance audit should continuously identify

deficiencies in information systems in need of correction for supporting the further
progress of the knowledge economy and information society in all countries.

Guidelines for the development and use of KNI in developing economies (the
example of the Commonwealth of Independent States member-states)

The issue of the development and use of KNI is very topical for countries that are on the way
to forming a knowledge-based economy and society. To increase the economic growth and
welfare of societies, such countries often create regional alliances and develop common
regional development strategies. For SAIs, above all, the organization and conduct of joint
audit-analytical activities needs common standards, agreed-upon procedures and evaluation
criteria and, most importantly, key indicators that should be determined jointly to satisfy the
overall goals and objectives of economic development.
To evaluate the effectiveness of socio-economic strategies in a developing economy,
guidelines are necessary. For example, the Commonwealth of Independent States member-
states developed Guidelines on the Use of Key National Indicators in Performance Audits.
The guidelines included all aspects of performance audit, including: the term of performance
audit; performance audit purposes; spheres of performance audit; performance audit steps;
methods of collection and analysis of information; preparation of an economic-
methodological basis of the audit; performance audit criteria determination; methodology of
indicators/KNI selection in performance audit; development of KNI system; definition of audit
evidence and received data analysis; preparation and distribution of the report on
performance audit results; monitoring of recommendations realization; and a glossary.
KNI can be considered as performance audit criteria by which outcomes of socio-economic
development strategies, government activities, socio-economic processes, and society
condition as a whole are evaluated. The most important KNI characteristic is comprehension
and interrelationships of goals, tasks, and indicators chosen or developed for the evaluation.
In developing economies, this involves the use of both international development programs
and national socio-economic development strategies.
In accordance with international commitments of CIS member states, it is recommended to
use indicators of Millennium Development Goals and sustainable development indicators as
KNI for the development goals included in national development strategies.
It is also recommended to use in CIS SAIs activity a system of the public financial
management (PFM) high-level performance indicator set developed by World Bank.
Of special significance is the fact that the Strategy of Economic Development of the CIS until
2020 (14.11.2008, Kishinev) has a set of main economic development indicators of CIS
member-states.
In order to harmonize methods of development with KNI sets and systems, it is proposed to
develop an indicator passport including: name of indicator, unit of measurement, periodicity
of estimation, characteristic, calculation methods, source of information, level of
disaggregation, variants of indicator.
Sets and systems of KNI are developing in accordance with economic development
strategies of CIS member-states. During joint audit activity they can be coordinated among
countries in the process of preparation of an economic-methodological basis of auditing.
 20

Final Statement

Modern methods of monitoring have neither prevented the global financial crisis nor fully
determined its effects. The inadequacy of existing regulatory instruments has given greater
urgency to the issue of developing and using KNI. Continued research in this direction is
intended to optimize the SAIs activities, to improve the quality of government actions and the
level of living standards. In addition, it is obvious that the selection of key indicators of socio-
economic development determines whether the country's choice of development goals is
adequate and appropriate.
In order to implement the above-mentioned initiatives in the post-crisis period, it is necessary
to combine the efforts of all experts, dealing with the issues related to the assessment of the
effectiveness of socio-economic development strategy implementation, as well as to bring
other interested parties on board.
At this stage, this white paper on KNI is mostly informational and accumulates the basic
principles and approaches in the development and use of KNI of socio-economic
development. This version of the white paper is aimed primarily at achieving mutual
understanding among all interested parties involved in the process of developing and using
KNI. One hopes that it can become a key document for SAIs and be a part of the general
development ideology with specific ways to achieve the identified goals through effective
methods of monitoring. In the future, it seems appropriate to complete this guide through
detailed evaluation of the issues.
Obviously, the continuous updating of knowledge in this area involves the use of modern
information technologies and the development of corresponding reference models. These
opportunities will ensure the transparency of the national assessment systems of socio-
economic development and the synchrony of changes reflected in KNI systems and methods
for their evaluation. In addition, new concepts and technologies can provide the evaluation of
contribution of the participants of socio-economic development processes in the final result.
Thus, the following approaches can be considered as basic:
 The selection of integrated indicators and indexes in terms of universal (transparent)
development models.
 The development of the multidimensional reference model in accordance with
transnational and national development goals.
These approaches are not alternatives, but are interrelated and interdependent.
The use of the concept of capabilities management, which has been developed recently,
seems to be efficient in this regard. Measuring progress by assessing the capabilities of
 21

development involves assessing the effectiveness of the socio-economic development

management system in general, including objects, processes, and governing bodies.
The INTOSAI Working Group on KNI considered a possible architecture of an information
reference model, which includes indicators of strategic goals achievement, key indicators of
development capabilities, indicators of key assets, and indicators of risk and capacity of
governing bodies. In this case, one may assert that the status of total economic capability
and strength is the object of the assessment.
In this reference model, key capabilities can be determined. These capabilities characterize
the ability and readiness of a system to achieve strategic goals, the outsourcing of which is
impractical. Such basic capabilities can be defined as sovereignty, security, competitiveness,
socio-cultural identity, vitality, satisfaction, etc.
In its turn, the level of socio-economic capabilities is determined by the presence,
distribution, and use of such assets as key system resources, including the possibility of
human capital, material and non-material assets, and ongoing processes.
Possibilities to assess the effectiveness of the national assets management in order to
concentrate them at the right time and in the right place for ensuring the competitiveness,
security and sustainability of development processes require developed network models.
Important factors are the accountability and assessment of the timeliness and adequacy of
governing bodies’ response to threats and risks of socio-economic development. In this
regard, KNI should also include indicators of threats and risks.
The selection of KNI that adequately reflects the object’s state is an extremely difficult task
that’s why it’s necessary to ensure the maximum use of modern information technologies
and resources. In addition, new concepts and IT-technologies can provide the measurement
and assessment of the final goals, including the assessment of the goal-setting and
architecture.
Knowledge bases, developed by key international organizations, including OECD, the World
Bank, the UN, the IMF, and the Davos Forum, are significant information resources. Within
the framework of the INTOSAI project on KNI, with the support of OECD, access to these
knowledge bases is provided. This access includes the possibility of analysis of presented
data on KNI.
Furthermore, the Working Group developed and tested a number of information technologies
for the efficiency estimates visualization of data on KNI. Implementation of this project will
ensure taking into account the interests of an SAI in the process of developing the
performance indicators system that would ensure transparency, objectivity, and
methodologically elaborated KNI and, in general, will affect the growth of professionalism of
the SAIs. In addition, through performance auditing with the use of KNI, SAIs will help to
improve the nation's economic and social policies, advising on implementation of
commitments.
It is important to note that by the joint efforts of all interested parties, it is possible to reach
one of the main goals of the Working Group on KNI and INTOSAI as a whole—improve the
effectiveness of the assessment of socio-economic development strategy implementation in
INTOSAI member states.
 22

Annex A

Glossary of terms

Accessibility reflects the ready ability to locate and access data, including the suitability of
the form in which the data are available, the media of dissemination, and the availability of
metadata and user support services.
Accuracy is the degree to which the data correctly estimate or describe the characteristics
that they are designed to measure. It refers to the closeness between the values provided
and the (unknown) true values. In general, the accuracy of the data is measured or described
in terms of the error or the potential significance of the error.
Assets include material values, money, debt claims etc., from which an organization expects
economic benefits in the future.
Capability refers to the ability to attain defined goals. Measuring progress through the
assessment of development capabilities involves assessment of the effectiveness of the
entire system of managing social and economic development, including its objectives,
processes, and systems. Capabilities, in turn, may be disaggregated into key material and
non-material assets.
Coherence of data reflects the degree to which they are logically connected and mutually
consistent. This implies that the same term should not be used for different concepts or data
items without explanation and that variations in methodology that might affect data values
should likewise not be made without explanation.
A composite indicator is built from a collection of individual indicators that are then
compiled into a single index on the basis of an underlying model of the multi-dimensional
concept that is being measured. A composite indicator measures multi-dimensional concepts
(e.g. competitiveness, the performance of an economy or environmental quality) which
cannot be captured by a single indicator. Some examples include:
 The Human Development Index, developed by the United Nations Development
Program, measures development by combining indicators of life expectancy,
educational attainment, and income into a composite human development index.
 The OECD composite leading indicators (CLIs) are designed to provide early signals
of turning points in economic activity. CLIs are calculated by combining component
series that cover a wide range of key short-term economic indicators, including, for
example, data related to economic activity, housing permits, production, and trade.
 23

Comprehensive key indicator systems pull together only the most essential indicators on
a range of economic, environmental, social, and cultural issues, as opposed to a group of
indicators on one topic. Comprehensive systems are only as good as the topical systems
they draw from. They can help to identify a jurisdiction’s significant challenges and
opportunities, highlight their importance and urgency, inform choices regarding the allocation
of scarce public resources, assess whether solutions are working, and make comparisons to
other jurisdictions. Some examples:
 German System of Social Indicators: Monitors status and changes in living conditions
and quality of life, covering 14 domains, including economic, environmental, social,
and cultural. Includes almost 400 indicators and 89 key indicators.
 UK Sustainable Development Indicators: Measures progress toward the
government’s sustainable development strategy in the areas of social progress,
economic growth, and environmental protection. Includes 15 headline indicators to
give a broad overview and 132 core indicators to focus on specific issues and identify
areas for action.
 European Structural Indicators: Indicators track progress toward strategic goals for
the economic, social, and environmental renewal of Europe, as detailed in the Lisbon
Strategy. The indicator system covers employment, innovation and research,
economic reform, social cohesion, and the environment. The EU reports on about a
dozen headline indicators, each consisting of a number of other supporting indicators.
Credibility of data refers to the confidence that users place in data products based on their
perceptions about the producer of the data. One important aspect is trust in the objectivity of
the data, which must be perceived as professionally produced in accordance with
appropriate statistical standards, having transparent policies and practices, and free of
manipulation or political pressure.
Data are specific quantitative and qualitative facts and figures obtained in the course of
information collection.
Data limitations are known problems with the data sources or the data that may be
identified by program evaluations, independent audits, information systems analyses, etc. If
significant, these limitations could lead to inaccurate assessment of goal achievement. Such
limitations might include:
 inconsistencies in data collection from location to location, from one time period to
another, or from one data source to another, when data from more than one source
must be combined to create a performance measure. Inconsistencies can arise when
standard procedures are not used or followed;
 inaccuracies due to imprecise measurement and recording; or
 incomplete data.
Data quality can be defined as “fitness for use,” a concept that includes a number of
attributes that contribute to the usefulness of the data from the perspective of the users,
specifically: relevance, accuracy, credibility, timeliness, accessibility, interpretability, and
coherence.
Dimensions and domains: Progress can be measured along a number of different
dimensions, which can be grouped into a smaller number of domains. Each dimension
reflects a basic kind of end or outcome, such as economic growth or human endeavor (e.g.,
education or health) or characteristics of our environment (e.g., air quality). A group of
related progress dimensions can be placed together in a domain (also sometimes called a
“pillar”). For example, indicators of national income and national wealth might both be
grouped under the economy domain.
 24

Footprints are composite indicators based on a calculation of the sum of all resources
required to provide specified goods and services. The Ecological Footprint by the World
Wildlife Fund, for example, calculates how much productive land and sea is needed to
provide the resources, such as energy, water, and raw materials, we use in our everyday
lives. It also calculates the emissions generated from the oil, coal, and gas we burn, and it
determines how much land is required to absorb our waste.
Frameworks for indicators display the choice of domains and dimensions to be included in
an indicator set and how they relate to one another. Frameworks are a tool to focus and
clarify the scope of an enquiry. They facilitate these tasks by delineating the dimensions
used to build up a particular concept and by creating a logical structure that illustrates how
these dimensions relate to one another.
Goals refer to the long-term aims of society, usually expressed in general terms. The term is
often used interchangeably with objective, although objectives can also be considered
subsets of a goal. A distinction is sometimes made between primary and secondary
objectives, with primary objectives relating directly to an outcome—for example, improving
public health—and secondary objectives being one of the means of achieving the primary
objective—in this case, for example, improving public health by providing safe drinking water.
Indicators are quantitative or qualitative statistics or measures that provide information on
the state of, or change in, a system over time, at either a national, regional, or local level.
The unemployment rate, infant mortality rates, and air quality indexes are examples. Some
indicators may be direct, that is, they measure what they say, for example, unemployment
rates. Other indicators may be indirect, or proxies. The number of patents granted, for
example, may be a proxy for measuring the degree of inventiveness. Some sources include
 Hundreds of indicators are used around the world, many of which are published by
OECD, the United Nations, and the World Bank, among other organizations.
Examples:
o The OECD Factbook comprises a set of more than 100 economic, social, and
environmental indicators, and may be viewed as a comprehensive reference.
(Seehttp://www.oecd-ilibrary.org/content/datacollection/factbook-data-
en;jsessionid=7wakbzj50w6d.x-oecd-live-02).
o The Millennium Development Goals were developed by United Nations
member states to reduce poverty, hunger, and disease, among other things.
About 60 indicators were developed to track progress against these goals.
(See
http://mdgs.un.org/unsd/mdg/Host.aspx?Content=Indicators/OfficialList.htm).
o The Public Expenditure and Financial Accountability Program, supported by
the World Bank, the European Commission and other government
organizations, developed a framework for assessing country public financial
management that includes a set of 28 high-level indicators related to budget
credibility, execution, and external audit, among others (See
https://pefa.org/content/pefa-framework).
Inputs represent the level of resources—material, energy, effort and money—used to
produce an output.
Interpretability reflects the ease with which the user may understand and properly use and
analyze the data. The degree of interpretability is largely determined by the adequacy of
definitions of concepts, target populations, variables, and terminology underlying the data.
 25

Key national indicators (KNI), sometimes referred to as “headline indicators,” define a core
set of information about the progress and position of a nation, selected from a range of
possibilities. There is no “right” number of indicators; how the balance is struck between
simplicity and breadth of coverage can vary widely. However, KNI are generally limited to
what society considers the “vital few.” While a set of indicators can include anywhere from a
few to dozens, any KNI set is not intended to be exhaustive, but rather, to provide a
summary picture of those conditions considered to be most important for the progress of a
nation. As is the case in defining progress, the process of selecting KNI is inherently political,
representing the aspirations and values of society.
Key national indicator (KNI) systems, which may be referred to as “suites of indicators,”
are organized efforts to assemble and disseminate a group of indicators that together tell a
story about the position and progress of a nation. Indicator systems collect information from
data collectors and package it into products and services for leaders, researchers, planners,
and citizens, among others. A KNI system generally includes social, economic and
environmental indicators of a nation to provide an overall picture of the country’s progress
and well-being. While many countries have indicators in one or another of these areas, a KNI
system can provide a comprehensive and balanced view, to help to ensure that one
dimension of progress is not advancing at the expense of another.
Material assets are those whose value is largely defined by physical features.
Non-material assets include an organization’s human capital and other non-monetary
assets.
Outcome indicators measure change that matters directly to a society, such as educational
attainment levels.
Output indicators measure change in the volume of products or services delivered, such as
the number of arrests or enforcement actions taken. These types of indicators are important
because outputs are usually produced in the hope of changing an outcome.
Performance management is a process of developing and using information on
performance to achieve a desired level of performance. Typically, a performance
management system consists of several elements:
 establishing the desired level of performance by setting goals and targets
 measuring performance through the use of one or more indicators
 reporting or communicating performance information
 comparing actual performance to the desired level of performance
 assessing the effectiveness of strategies in achieving goals and targets and taking
any necessary corrective actions
Performance measures are indicators, statistics or metrics that are used to gauge the
performance of an activity, process, or operating entity. Performance measures are also the
reference markers used to measure whether a goal is being achieved. To be able to assess
progress toward the achievement of performance goals, the measures used must be valid
and reliable. In order for measures to be valid and reliable, the data on which they are based
must be free from significant error, especially bias.
Pressure-State-Response indicators provide a framework for the presentation of indicators
(often environmental) arranged according to the pressures that human activities exert on an
area of concern, the state of the problem, and of society's responses. For example, in the
area of climate change, indicators could be the amount of CO2 released into the atmosphere
each year (pressure), average temperature rise (state); and money spent combating adverse
weather (response).
 26

Progress means that life getting better for a society, as defined by members of that society.
Progress may also be defined as success in attaining or nearing the goals that are
established through a political process or other type of civic engagement. Progress is multi-
dimensional and typically includes economic, social and environmental factors along with
other areas that people see as important to life (for example, culture, or the quality of
governance). Although progress implies change for the better, any assessment of progress
must also include assessment of regress.
Reference models are unitized reflections of standard elements and their relationships and
logical interconnections. In this sense, a KNI system may be considered a reference model
for measures of progress and position. A reference model allows assessment of KNIs
independent of the selection of indicators, such as GDP or Welfare Index.
Relevance refers to the degree to which the data serves to address the purposes for which
they are sought. Measuring relevance requires identification of user groups and their needs,
both of which can change over time. Relevance may be indirectly assessed by determining
whether there are processes in place to determine the views of users and the uses they
make of the data.
Reliability refers to the precision with which performance is measured.
Strategy is a course of action, or the means by which to achieve goals and objectives.
Developing a strategy includes identifying suitable points of intervention and ways of
ensuring the involvement of appropriate entities, considering the range of political, social,
economic, managerial, and technical factors that affect the strategy and defining the possible
constraints and ways of dealing with them.
Selecting indicators for an indicator system can involve different approaches:
 A bottom-up approach works from the grassroots, causing a decision to arise from
the joint involvement of a large number of people working together.
 A top-down approach has an executive decision maker or body that chooses the
indicators, although the choice of indicators might be based on consultation with
others.
Subjective well-being is a measure of how people feel about their lives or aspects of their
lives. It refers to a broad category of phenomena that includes people's emotional responses,
satisfaction with various domains, and satisfaction with life in general.
Sustainable development is defined as a development path along which the maximization
of human well-being for today's generations does not lead to declines in future well-being.
A target is an intermediate result toward the achievement of goals and objectives. A target
generally has a time horizon and is frequently, although not always, quantified. A target
related to public health might be to ensure that a certain percentage of the population has
access to safe drinking water by the year 2014. Frequently, targets follow a framework called
SMART: Specific, Measurable, Achievable, Results-oriented and Time-bound.
Timeliness of data reflects the length of time between their availability and the event or
phenomenon they describe, considered in the context of the time period that permits the
information to be of value and still acted upon.
Topical indicator systems involve specific or related sets of issues, such as health,
education, public safety, employment, or transportation. They also form the foundation of
information resources for the general public, the media, professionals, researchers,
institutions, leaders, and policymakers. An example:
 27

 OECD Social Indicators: OECD’s Society at a Glance reports present a variety of

social indicators that provide perspectives on self-sufficiency, equity, health, and
social cohesion.

Validation is the testing of data to ensure that no error creates significant bias, to avoid
affecting conclusions about the extent to which performance goals have been achieved.

Validity is the extent to which a measure adequately represents actual performance.

Verification is the checking or testing of performance data to reduce the risk of using data
that contain significant errors.

Well-being: Assessments of societal progress often focus on the well-being of society, or the
condition or state of being well, contented, and satisfied with life. Dictionary definitions differ,
but notions of prosperity, health, and happiness generally figure. The term, “quality of life,” is
sometimes also used to indicate the condition of social well-being.
 28

Annex B

Countries and international organizations’ experiences

in the development and use of key indicators

Countries experience

Australia
 Website of the Australian Bureau of Statistics. Measures of Australia's Progress,
http://search.abs.gov.au/s/search.html?query=measures+of+Australia+Progress&coll
ection=abs&form=simple&profile=_default_preview
Canada
 Website of the Canadian Index of Wellbeing, https://uwaterloo.ca/canadian-index-
wellbeing/
China
 Website of the National Bureau of Statistics of China, http://www.stats.gov.cn/english/
Finland
 Website on Finland Indicators, http://www.findicator.fi
Ireland
 Website of the Central Statistics Office of Ireland. Measuring Ireland’s progress,
http://www.cso.ie/en/statistics/measuringirelandsprogress/

Poland
 Website of the Central Statistic Office of Poland. Sustainable Development Indicators
for Poland,
http://stat.gov.pl/en/topics/other-studies/other-aggregated-studies/sustainable-development-
indicators-for-poland-2015,3,2.html

 Poland - macroeconomic indicators,

http://stat.gov.pl/en/poland-macroeconomic-indicators/

 Website of the Ministry of Regional Development of the Republic of Poland. An

Integrated Approach to Development in Poland,
http://mttp.pl/pobieranie/MRR_Polityka_Rozwoju_ANG.pdf
 29

Switzerland
 Website of the Swiss Federal Statistical Office. The MONET Indicator System,
https://www.bfs.admin.ch/bfs/en/home/statistics/sustainable-development.html
United Kingdom
 Website of the United Kingdom Department for Environment, Food and Rural Affairs.
Measuring progress: sustainable development indicators,
https://www.gov.uk/government/collections/sustainable-development-indicators

USA
 Website of nonprofit organization, The State of the USA, http://stateoftheusa.org/
 Website of the U.S. Government Accountability Office. Informing Our Nation:
Improving How to Understand and Assess the USA’s Position and Progress, GAO-
05-1 (Washington, DC, November, 2004), http://www.gao.gov/products/gao-05-1

International organizations’ experiences

European Union
 Website of the Eurostat. Europe 2020 indicators,
 http://ec.europa.eu/eurostat/web/europe-2020-indicators/europe-2020-strategy
G-20
 Website of G-20.Communiqué Meeting of Finance Ministers and Central Bank
Governors, Washington, D.C., April 14-15, 2011,
http://www.g20.org/
INTOSAI
 Website of the INTOSAI Working Group on KNI,
http://www.ach.gov.ru/en/activities/international-activities/intosai-working-group-on-key-
national-indicators/
Organization for Economic Cooperation and Development (OECD)
 Website of the OECD. Key Environmental Indicators,
http://www.oecd.org/env/indicators-modelling-outlooks/

 Website of the OECD. Society at a Glance 2011 - OECD Social Indicators,

http://www.oecd.org/social/society-at-a-glance-19991290.htm

 Website of the OECD. Developing societal progress indicators: a practical guide,

http://www.oecd.org/officialdocuments/displaydocumentpdf?cote=std/doc(2010)6&do
clanguage=en

 Website of the OECD. Handbook on constructing composite indicators,

http://www.oecd.org/document/9/0,3746,en_2649_34349_41752777_1_1_1_1,00.ht
ml
 Website of the OECD. Better Life Index, http://oecdbetterlifeindex.org/

 Website of the OECD. Quality Framework and Guidelines for OECD Statistics,
 30

http://www.oecd.org/dataoecd/26/42/21688835.pdf

 Website of the OECD. Sustainable Development Glossary,

http://www.oecd.org/std/qualityframeworkforoecdstatisticalactivities.htm

 Website of the OECD. A Framework to Measure the Progress of Societies, Giovanni,

E., Hall, J., Morrone, A., & Ranuzzi, G. OECD Working Paper, http://www.oecd-
ilibrary.org/economics/a-framework-to-measure-the-progress-of-societies_5km4k7mnrkzw-en
United Nations
 Website of the United Nations. Indicators of Sustainable Development,
http://www.un.org/esa/sustdev/publications/indisd-mg2001.pdf

 Website of the United Nations Development Program. Human Development Index,

http://hdr.undp.org/en/statistics/hdi/

 Website of the United Nations. Millennium Development Goals,

http://www.un.org/millenniumgoals/
World Bank
 Website of the World Bank. World Development Indicators,
http://blogs.worldbank.org/opendata/2016-edition-world-development-indicators-out-three-
features-you-won-t-want-miss

 Website of the World Bank. World Development Report,

http://www.worldbank.org/en/publication/wdr/wdr-archive

 Website of the World Bank. Global Monitoring Report,

http://www.worldbank.org/en/publication/global-monitoring-report

 Website of the World Bank. Public Expenditure and Financial Accountability,

http://documents.worldbank.org/curated/en/483831467998500044/Public-expenditure-and-
financial-accountability-PEFA-assessment-Nepal-PFM-second-performance-assessment-as-
of-FY2013-14

 Website of the World Bank. Adjusted Net Savings,

http://siteresources.worldbank.org/INTEEI/1105643-
1115814965717/20486606/Savingsmanual2002.pdf
World Economic Forum
 Website of the World Economic Forum. Global Competitiveness Index,
http://reports.weforum.org/global-competitiveness-report-2015-2016/competitiveness-library/
 31

Annex С

Principles for the Development and Use of Key National Indicators

INTRODUCTION

Key National Indicators (KNI) help countries evaluate and align their strategies and programs
for socio-economic development, and other meaningful results. The overall goal of a KNI
system should be to trigger better results from public administration. The INTOSAI Working
Group on KNI presents the following principles to help public administrators and SAIs
develop, use, and assess KNI. The first part of this document focuses on the main conditions
necessary to organize the process of KNI development and use. The second part describes
the role of SAIs in this process.

BASIC CONDITIONS FOR KNI DEVELOPMENT

Leaders at all levels of government are seeking creative ways to use their resources more
efficiently and effectively to serve the public. Attention to promising evidence-based practices
has increased, as have efforts to eradicate inefficiency within the system. Outcome
management is all about planning, managing and achieving the intended outcomes of an
initiative or a program in the public sector. It is all about having the same focus and discipline
in attaining these outcomes around delivering the capability and the systems in an on-time
and on-budget manner.

1. To develop a KNI system, a country needs to have public and private organizations
providing various types of support:

 Legal and regulatory requirements: A legal foundation that establishes the KNI
system and describes its use is often necessary. This legal foundation may be
statutory. It is important to differentiate between the legislation and regulations of
oversight organs of state and the legislation and regulations of line ministries as the
first group define macro policies and frameworks related to planning, budgeting and
programme implementation. While the second group - line ministries develop legal
and regulatory frameworks related to their core business (such as Patient Protection
and Affordable Care Acts in the United States), or a special regulatory document that
reflects the sphere of responsibility of all actors in the KNI process for that ministry.
The legal foundation is important to guarantee the legitimacy of activities associated
with the KNI process.

 Government-wide planning initiatives/requirements: This can set the

initiatives/requirements for departmental plans and reports. It also allows
departments to link their performance to the achievement of governmental strategic
priorities. This approach brings staff from the finance departments, policy offices and
cabinet executives together to identify priority outcomes in each of governmental
prioritised “result areas”. All resources for planning, budgeting, monitoring and
evaluation are grouped together to ensure one single line of performance and
reporting using one core set of performance indicators. Such an approach also allows
departments to include international objectives they need to report on as part of their
 32

programme delivery. This approach promotes alignment of planned achievements as

outlined in outcomes based approaches and catered for in budgets.

 Institutional: The stakeholders responsible for the development and use of a KNI
system must be identified. When identifying stakeholders, it is necessary to take into
account historical, political, institutional and cultural factors specific to the country.
The organizational structure responsible for the KNI system development may be a
state authority, a research and development institute, an institution engaged in the
issues of the accountability of public policy, a statistical institution, or a network of
several of the above-mentioned structures interacting on the basis of a special
agreement. The roles and responsibilities of each of the stakeholders need to be
clear to all participating government departments.

 Informational and methodological: Organizations that have experience and

information about developing and using a KNI system should be encouraged to share
that experience and information. To maintain the ongoing sharing of experiences and
information, it seems appropriate to use the Knowledge Base on KNI that was
developed in the framework of the INTOSAI Working Group on KNI. This tool may be
used to work simultaneously with information from various sources (IMF, World Bank,
OECD, etc.) in order to analyze the dynamics of economic and social processes by
periods, countries, regions, and sectors. In addition, the Knowledge Base allows all
INTOSAI members to compare economic, social, and political experiences of the
countries studied in relation to different socio-economic variables, in order to measure
their own progress as well as their position relative to others.

 Standardizing Program Governance

Most countries follow programmatic approaches in budgeting and planning. An

international standardized approach to program management, will contribute to the
development of a standardized approach to performance indicators.

Program governance is the aspect of the discipline that creates both the structure and
practices to guide the program and provide senior-level leadership, oversight and
control. Strategically, it encompasses the relationship between the oversight effort
and the enterprise's overall business direction. It also encompasses all the decision-
making roles and responsibilities involved in executing the program effort.

 Financial: Public and/or private funding of a KNI system’s development and use must
be sufficient and sustainable. This is important to allow the ongoing use of available
data and acquisition and analysis of new data to improve KNI at the national and
international levels.

2. When developing and selecting the indicators themselves, a country needs to ensure that
these basic requirements of a KNI system are met:

 Complexity/Comprehensiveness: The KNI system must cover the key areas that
are relevant to society and must allow for new indicators to be added as needed;

 Reliability: The KNI system must be created and indicators selected using stable,
consistent, accurate, and reliable data and tools to ensure that the system reflects the
society accurately.
 33

 Continuity: Information that helps determine the evolution of the issue being
measured must be available.

 Systematization: Consideration must be given to how the indicators will be used to

assess and improve government’s performance and contributions to socio-economic
development and other results.

 Validity: A country must be able to demonstrate that indicators actually measure

what they purport to measure.

 Methodological soundness: The data on each indicator must be as detailed as

possible, considering time, geographical, and other constraints.

KEY PRINCIPLES OF SAI ACTIVITY

1. In selecting and using KNI, SAIs should adhere to the following general principles which
are based on selected countries’ experiences with KNI and the appropriate roles and
responsibilities of SAIs:

Independence: To ensure that a SAI’s involvement in the consulting, analysis and

monitoring of the development of key national objectives, indicators, and implementation
targets is not viewed as audit impairment or as contributing to any potential design flaws, a
SAI must be operating independently under a valid mandate.

Objectivity and impartiality: In consulting, analyzing, and monitoring the development and
use of KNI, SAIs must be objective and impartial in order to maintain credibility as part of the
KNI system.

Professionalism: Professional knowledge and experience in political and methodological

issues related to KNI is necessary for the relevant SAI staff involved.. Knowledge and
training regarding KNI use in audit activities should be maintained and continuously
improved. The SAI staff has to acknowledge the fact that single indicators or statistics do not
necessarily reveal the effectiveness of policy interventions, due to various confounding
factors. Multivariate statistical analysis is often needed in order to separate the effects of
policy interventions.

Transparency and accountability: One of the main responsibilities of audit institutions is

maintaining the transparency and accountability of all audit activities. SAIs should therefore
plan, assess and develop appropriate audit methods and approaches when auditing or
assessing KNI’s. In addition, SAIs should also help ensure that all KNI-related activities are
as transparent as other similar government activities in that particular country.
 34

2. Within the broader framework and requirements of the ISSAI’s13 as well as the legal
mandate of the SAI, SAIs can consider the following goals:
 Enhance SAI assessments of government efficiency and effectiveness by undertaking
strategic assessments using KNI’s to inform assessments of government’s performance
and related performance information.
 Help INTOSAI member countries improve their audits of socio-economic development
activities by helping to develop universal approaches to using KNI in audits.
 Help improve knowledge- and experience-sharing on developing and using KNI in
countries that do not currently have a KNI system.
 Promote the establishment of INTOSAI as an authority on how to develop KNI and use
them for auditing national progress.

3. Within the broader framework and requirements of the ISSAI’s as well as the legal
mandate of the SAI, SAIs can consider the following tasks:
 Assess the credibility, reliability, objectivity, integrity, independence, and comparability of
KNI used by governments and the institutions that provide information for the calculation
of KNI values.
 Expand KNI use in audit activity. In particular, performance and strategic audits may
allow KNI to be used as effectiveness criteria to assess the results of development
strategy implementation and other activities of governments, socio-economic processes,
and the state of society as a whole.
Add value by the audit (and possible certification) of government performance management
information as a separate discipline or integrated into its core audit processes, beyond
traditional audit mandates.
 Increase public institutions’ awareness of and involve them in the programs associated
with KNI selection and the events associated with KNI use in audit activities.
 Use the latest tools and techniques in economic and statistical assessment, sociological
surveys, mathematical and software support, and means of visualizing results along with
conventional methods to work with KNI.
 Provide access to KNI-related information and guidance documents developed by
international organizations in the frameworks of various initiatives.
 Promote the broadest possible knowledge- and experience-sharing among national SAIs
on the issues of using KNI in audit activities.

4. SAIs can facilitate accountability and knowledge-sharing in a few ways:

 Regularly publish special reports or other documents on SAI outputs and

outcomes (results): Regular reports help SAIs influence national economic
development and enhance their own authority and value. Over time, regular
reports can help optimize the KNI system and introduce adjustments into the
interactions among system stakeholders.

 Accumulate and disseminate knowledge obtained as a result of KNI use:

Collecting and distributing this information can help ensure that the information is
continuous, which can be especially important when trying to use information to
solve a problem. The use of KNI in audit activities should not be limited to
obtaining traditional statistical information. The analysis based on KNI and the
monitoring of KNI dynamics, if regularly updated, could make it possible to

13
SAI’s can consider providing value added services within the context as per ISSAI 5000 - 5999
 35

significantly improve effectiveness in the use of public resources and in the

country’s strategic planning system.

 Share existing national and international experiences with developing and

using KNI in audit activities and take global trends into account: Sharing
experiences nationally and internationally and being aware of global trends in
social and economic development will help SAIs learn about KNI developments
and how KNI can be used effectively in their audits. In turn, this promotes
continuous, sustainable, and topical KNI use in audit activities and enables SAIs
to promote best practices in governance with transparency and accountability.

5. The audit or assessment of the development and use of KNI should be based on or
consistent with the Fundamental Auditing Principles (ISSAI 100) of the International
Standards of Supreme Audit Institutions (ISSAI’s). Each stage are outlined below:

 First stage: Audit or assessment planning which include aspects such as

engagement acceptance, background information-gathering, understanding the client,
determination of materiality, risk assessment and development of the audit plan.
SAIs may undertake assessment of their countries’ KNI efforts in such areas as the
following:
o Identification of KNI based on priorities specified in published strategies and
development programs.
o Identification of KNI used by other INTOSAI member countries for assessment of
the same sector.
o Verification of the quality of selected KNI, including an assessment of each
indicator’s strong and weak points.
o Determination of whether selected KNI are quantitative or qualitative; interval or
real-time; individual or aggregated.
o Determination of how selected KNI should be disaggregated to best monitor the
performance of the particular process or processes at hand—the overall
performance of society is composed of several individual parts and indicators
should reflect that.
o Determination of how frequent reports on the dynamics of selected KNI should be.
o Identification of the information sources to be used for submitting data on KNI,
taking into account that the data should be analytically substantiated, measurable,
topical and relevant.
o Preparation of the KNI data collection plan.
o Continuous improvements such as refining the KNI set, taking into account the
identified limitations and the data collection plan and repeat the steps above for
several cycles with varying levels of disaggregation and monitoring frequency to
form a shorter list of indicators and data sources.
o Acknowledgment that selecting KNI that best reflect reality is a complicated
process that takes time. The legitimacy of the final result will depend on the
effectiveness of the responsible agency’s interaction with stakeholders and the
degree of their involvement in this process.
o Assurance that the indicator selection process accounts for political implications
and existing constraints. In general, given the optimal level of disaggregation, it is
preferable to have a small number of indicators of good quality answering the
most important questions that are easily measurable in view of reality.
o Assurance that selected indicators reflect not only the changes that happen in a
particular place at a particular time, but also the important factors affecting the
country’s economic performance.
 36

 Second stage: Audit or assessment execution, , including main research, design of

audit or assessment programs, obtaining and evaluating audit evidence, audit testing
(which can include sampling) and reaching a reliable audit conclusion. SAIs may
undertake assessment of their countries’ KNI efforts in such areas as the following:
o Using selected KNI to identify audit findings (for example, the degree to which
government strategies and programs are aligned with and contribute to
improvements in selected KNI) and express the audit opinion.
o Assessing the KNI system as a whole including summarizing and analyzing data
and information to assess whether selected indicators should be reviewed and
methods should be optimised.
o Audit (and possible certification) of systems used to generate national
performance management information.

o Audit (and possible certification) of the credibility of government performance

management information.

o
 Third stage: Reporting audit results by communicating audit results for a broad
audience by preparing printed reports, publishing on the Internet, or presenting
results in person. SAIs may undertake assessment of their countries’ KNI efforts in
such areas as the following:
o Ensure that audit results that were obtained using KNI are accumulated and
disseminated.

 Fourth stage: Monitoring the implementation of audit recommendations.

o Ensure that knowledge on the use of KNI in audit activities is accumulated and
disseminated.

List of source documents used in the preparation of the Principles for the
Development and Use of Key National Indicators.

1. Europe 2020. A strategy for smart, sustainable and inclusive growth, EU, 2010.
2. Global Competitiveness Report 2010-2011, World Economic Forum, 2010.
3. Global Monitoring Report 2010, the MDGs after crises, the World Bank, 2010.
4. Human Development Report 2010 —20th Anniversary Edition. The Real Wealth of
Nations: Pathways to Human Development, UN, 2010.
5. Lisbon Strategy 2000-2010, EU, 2000.
6. Millennium Development Goals Report 2010, UN, 2010.
7. Society at a Glance 2011 - OECD Social Indicators, OECD, 2011.
8. World Development Indicators 2011, the World Bank, 2011.
9. World Development Report 2011, Conflict, Security, and Development, the World Bank,
2011.
10. Декларация Тысячелетия ООН, Саммит тысячелетия ООН, 2000.
11. An approach to the 'Canadian Index of Wellbeing. A report of the 'Canadian Index of
Wellbeing, 2010.
 37

12. Canada’s Performance 2006–07. Annual Report of The President of Treasury Board of
Canada to Parliament. 2007.
13. HR 3590, Patient Protection and Affordable Care Act, Title V. Section 5605, USA, 2010.
14. Key National Indicators. Experiences of other national and subnational systems offer
insights for the United States, US GAO, 2011.
15. Measuring Ireland’s progress 2009, Central Statistics Office of Ireland, 2010.
16. Measures of Australia's Progress, Australian Bureau of Statistics, 2010.
17. Measuring progress: sustainable development indicators 2010, DEFRA, United Kingdom,
2010.
18. Outline of the Twelfth Five-Year Plan for National Economic and Social Development of
the People’s Republic of China, 11th National People’s Congress, 2011.
19. Report “How are Canadians Really Doing?”, The Institute of Wellbeing, 2009.
20. Report of the Commissioner of the Environment and Sustainable Development, Canada,
2003.
21. Adjusted Net Saving: A Manual, the World Bank, 2002.
22. Developing societal progress indicators: a practical guide, OECD, 2008.
23. Guidelines for developing a national programme of indicators of sustainable
development, UN Division of Sustainable Development, 2001.
24. Handbook on Constructing Composite Indicators. Methodology and user guide, OECD,
2008.
25. Indicators of Sustainable Development: Guidelines and Methodologies, UN Division of
Sustainable Development, 2001.
26. Key environmental indicators, OECD, 2008.
27. Kirk Hamilton. Genuine Saving as a Sustainability Indicator // The World Bank
Environment Department. Environmental Economics Series. October 2000.
28. The Little Green Data Book, the World Bank, 2008.
29. The World Bank’s Genuine Savings Indicator: a Useful Measure of Sustainability? the
World Bank, 1999.
30. United Nations, Indicators of Sustainable Development: Framework and Methodologies,
UN, 1996.
31. Public Expenditures and Financial Accountability. Management of public finances.
Performance evaluation system, the World Bank, 2005.
32. World Economic Outlook. The two rates of growth contradictions. Unemployment,
commodity prices and capital flows, IMF, 2011.
33. Agenda XXI, United Nations, 1992.
34. Guidelines for the poverty monitoring. The choice of indicators, the World Bank, 2004.
35. White Paper on KNI. INTOSAI Working Group on KNI 2013.
 38

36. Guidance "Indicators for monitoring progress towards the Millennium Development
Goals: definitions, rationale, concepts and sources", United Nations, 2003.
37. Global Competitiveness 2010-2011. Report of the World Economic Forum in Davos,
Switzerland, in January 2011, "Working translation . Center for the situational analysis of
the Accounts Chamber of the Russian Federation, Moscow, 2011.
38. The research report: "The development of conceptual proposals for the formation of the
INTOSAI Glossary of key national indicators," Research Institute of the joint venture,
2009.
39. Tarasova N. Indices and Indicators for Sustainable Development / NP Tarasova, EB
Kruchinina / / Sustainable development: nature - society – human. The International
Conference - Moscow, 2006.
40. Indonesia's SAI website, www.bpk.go.id/en.
41. World Bank website, www.worldbank.org.
42. The World Forum website, www.weforum.org.
43. Government Accountability Office website (USA), www.gao.gov.
44. Department for Environment Food & Rural Affairs website (UK), www.defra.gov.uk.
45. EU website, www.europa.eu.
46. Canadian Index of Wellbeing website, http://www.ciw.ca.
47. INTOSAI website, www.intosai.org.
48. UN website, www.un.org.
49. The State of the USA website, www.stateoftheusa.org.
50. Office of the Auditor General of Canada website, www.oag-bvg.gc.ca.
51. OECD website, www.oecd.org.
52. Finland's performance website, www.findicator.fi.
53. United Nations in the Russian Federation website, www.unrussia.ru.
54. Public Expenditure and Financial Accountability Program Website, www.pefa.org.
55. The United Nations Development Program website, www.undp.org.
56. INTOSAI Working Group on KNI website, www.kniknowledgebase.org,
http://www.ach.gov.ru/en/activities/international-activities/intosai-working-group-on-key-
national-indicators/.
57. The EU's statistical office website, www.ec.europa.eu/eurostat.
58. Australian Bureau of Statistics website, www.abs.gov.au.
59. Accounts Chamber of the Russian Federation website, www.ach.gov.ru/en/.
 39

Annex D

SAI's role in development and use of key indicators for research

and development (R&D) evaluation

Timo Oksanen & Ville Vehkasalo, National Audit Office of Finland14

1.10.2012

14
Correspondence to ville.vehkasalo@vtv.fi. The views expressed are those of the authors, not the
NAO of Finland.
 40

1. The starting points of the Finnish R&D –policy

“Public investments in research and education have traditionally been seen as the key
drivers for success of the Finnish economy and society, and the current economic down turn
has nothing but reiterated this viewpoint. …The fact that there prevails a rather harmonious
view about the best of science and best of Finnish society and its citizens, is reflected in
several ways that the place of science in society is constructed through public policies and
debates.” 15

In line with the MASIS-report, we can therefore define three basic principles or practices of
the Finnish education, research, and innovation policy:

1. Continuing consensus about science and innovations as a strategy for national success
and prosperity is a strong historical trajectory in Finland.
2. Recent and past science barometers indicate that the level of public knowledge about
scientific facts is high, and Finns have high trust in the institutions of science and technology.
3. The Finnish science and technology policy culture can be characterized as exclusive-
corporatist in the sense that central stakeholder and interest groups, in addition to scientific
advisors and state officials, are regularly consulted in the regulatory processes, while citizens
as interested individuals or as members of civil society organizations do not have such a
formalized role.16

2. Possibilities of developing main indicators of R&D in general?

The key challenge of R&D indicators is their ability to help us see complex things, such as
innovation, in a different way. First we have to identify the risks and general “fuzziness”
between R&D inputs, processes, and outputs, and also the risks concerning education and
innovation. This is a fundamental challenge in developing valid and relevant indicators for
R&D. Problems in indicators and the problem of complexity also seem to be quite parallel. To
put it loosely: the bigger the fuzziness and complexity at the R&D field is, the more difficult it
is to develop valid, reliable, relevant, and comprehensive indicators in this field.

According to Patton (2011) there are two main dimensions in the complexity of knowledge:
the quantity and quality of knowledge and the multitude of public values and goals17. Figure 1
illustrates these issues.

15
MASIS–report 2011, 16.
16
Pelkonen, 2008.
17
Patton 2011, 94 and 109-110.
 41

Figure 1. Complexity as a problem of knowledge and multitude of values and actors as a

main challenge of indicators18

Far from

Socially Complicated Chaos

Massive avoidance
Build relationships Zone of Complexity

create common ground

Education and Research and
Agreement Innovation indicators
Innovation indicators
Research indicators
Education indicators
Simple Technically Complicated

Plan, control Experiment, coordinate expertise

Certainty
Close to Far from

In the simple situation, there is a high agreement on values and goals and our knowledge
about causality is also on a high level. In this situation, we are near the origin of the diagram.
If there are conflicting values and our knowledge about causality is limited, we are moving
away from the origin towards more complicated, complex and even chaotic environments.
The third factor besides complexity is the number of actors (e.g. organisations, ministries,
institutions etc.).

In figure 1 we see that aiming to connect education, research, and innovation indicators
together is much more difficult and complex than concentrating only on one of these fields. In
a nutshell, the complexity of developing indicators increases as the field or the scope of the
field expands because the multitude of values and gaps of knowledge will also increase as
the field or scope expands.

3. Possibilities of developing R&D indicators in the Finnish context

In the field of R&D it seems that the starting points of the Finnish R&D –policy (chapter 1) are
favourable for developing the main indicators of R&D. In figure 1 high consensus about
values means agreement along the Y-axis of the diagram and moving towards the origin.
Therefore we should be able to develop a common understanding as a ground for the
indicators of R&D. On Y-axis we are quite close to the origin.

18
For a detailed description of the figure see Patton 2011, 90.
 42

The major problem lies in our limited knowledge concerning the causality between R&D
inputs and outputs and the impacts of education, research and innovation, whether we study
them separately or (especially) when studied jointly. Our certainty about the outcomes and
especially impacts of R&D is rather incomplete, and in figure 1 we are far from origin along
the X-axis. As Lena Tsipouri put it in her keynote speech at the London EES conference
2006,
 “The relationship between R&D-inputs and outputs remain largely a black box".
“However, the original black box is getting more transparent: science-based and
radical innovations are captured by patents, publications, citations etc. But when it
comes to more difficult issues (generation of externalities), we still have to learn more
than what we know”.
 In the field of R&D and innovation, "productivity is
not a linear or other function of the inputs but it depends on the structures, which
allow (or not) exploiting these inputs: absorptive capacities, scale, capacity utilization,
etc. Hence the challenge for a good evaluation is to capture the right parameters in
any given time and place”. This demands more qualitative approaches of evaluation.

In this situation the potential value of developing Finnish R&D-indicators can firstly be found
in their capacity to build common ground for fruitful dialog between different interest groups
and organisations in the field of education, research and innovation than in their ability to
measure R&D and innovation with validity and reliability. Secondly, the value of indicators
lies in their possible capacity to increase and integrate our understanding and knowledge of
R&D. A third potential value of R&D indicators lies in their ability to integrate different fields
(research, education, housing, traffic, innovation etc.) that are closely intertwined, rather than
measuring them separately.

4. The current measurement of R&D in Finland

According to a recent report19 by Statistics Finland, the annual increase in research and
development expenditure was 185 million euros in 2010. The majority of the growth, 77 per
cent, was directed to the higher education sector. Most of this growth came from the growth
in external funding. The changes in research funding by the state remained minor.

Public sector R&D expenditure increased by 36 million euros. Business enterprises'

expenditure on product development remained practically unchanged. The expenditure on
research and development performed in Finland reached 6.97 billion euros in 2010. The
R&D expenditure of business enterprises amounted to 4.85 billion, expenditure of the higher
education sector to 1.42 billion and the rest of the public sector to 692 million euros. The
share of the higher education sector in total R&D expenditure exceeded the limit of 20 per
cent, while the share of business enterprises dropped to nearly 70 per cent.

Findicator is a new Finnish databank which aims to produce a comprehensive approach to

assess the state and quality of Finnish society. According to the official description20 of the
databank, Findicator is an interesting collection of indicators on social progress in a user-
friendly form. The service has been set up together with users and information providers and
it brings together statistics and indicators that are already available in different formats.
The service is targeted at everyone needing up-to-date, reliable information on social
progress in their work or other activities: decision makers, public servants, specialists,
teachers, journalists, citizens. The indicators have been selected in consultation with user
groups and data providers. The set of indicators will be updated as appropriate.

19
Research and development 2010, Statistics Finland.
20
see http://www.findicator.fi/en/info.
 43

Findicator was opened in 2009 and has been widely used ever since. The process of
selecting the indicators went through four phases:
1. Survey of national and international sets of indicators and identification of the most
common indicators: When Findicator was in the planning stage, a number of national
and international sets of indicators measuring social or sustainable development were
examined. This analysis resulted in a list of some 100 indicators for social progress,
and these were then grouped according to theme.
2. Consultation with potential users:
The indicator list was further defined on the basis of feedback received from 15
individuals working in close contact with political decision making (including Members
of Parliament, their assistants, public servants from parliamentary group offices,
information specialists, etc.).
3. Consultation with experts:
Experts from a variety of ministries and branches of government along with
researchers and contacts from statistics providers were asked to comment on the list
as modified on the basis of user requests. Sources and their ability to make data
available for the service were also looked into.
4. Content production:
The list of indicators served as the basis for the online service. As the process of
compiling the statistics got under way, the indicator set was further refined according
to the actual availability of data.

Our own interpretation about its utilities and problems as a source of main national R&D
indicators is as follows. We think it is fair to say that the Findicator’s information content
seems to be quite narrow in this respect. There are only two indicators in the Findicator that
measure R&D outcome and effort:
 Time series of patent applications, fig. 2
 Time series of R&D expenditures, by sector, fig. 3.
Therefore at present the Findicator framework gives us only a brief introduction about the
current RDI development.

Figure 2. Patent applications in Finland, 1972 – 2010

5000

4500

4000
Foreign
number of patent applications

3500

3000

2500

2000
5. Development projects Finnish businesses
1500
VINDI
1000
http://www.aka.fi/Tiedostot/Tiedostot/Julkaisut/06_08%20VINDI.pdf
500
Finnish private
0
1970 1975 1980 1985 1990 1995 2000 2005 2010
 44

Figure 3. R&D expenditure by sector, 1971 - 2010

8000

7000

6000

5000
million euros

universities
4000 government+npo
business

3000

2000

1000

0
71

73

75

77

79

81

83

85

87

89

91

93

95

97

99

01

03

05

07

09
19

19

19

19

19

19

19

19

19

19

19

19

19

19

19

20

20

20

20

20
5. The main development projects of the indicators of R&D in Finland

Starting in the 1980’s and onwards, several efforts have been organised by the Finnish
government and other institutions in order to achieve a comprehensive framework for the
evaluation and measurement of research, development, education, and innovation activities.
Next list of internet links includes only the main development lines and specifically Finnish
efforts. The list is followed by a case study of one of these projects. Finland has also been
active in co-operation with EU and OECD indicator development projects.

METHODS:
http://www.aka.fi/globalassets/awanhat/documents/tiedostot/julkaisut/9_06-methods-for-
evaluating.pdf

UNIVERSITIES:
http://www.minedu.fi/OPM/Julkaisut/2007/Yliopistojen_yhteiskunnallinen_vuorovaikutus.html
?lang=fi&extra_locale=en

INDICATORS:
http://www.stat.fi/til/ttt_en.html
https://www.tekes.fi/globalassets/julkaisut/better_results_more_value.pdf

TECH BAROMETERS:
https://www.tek.fi/fi/uutishuone/tutkimukset/tekbaro

GOVERNMENT:
http://www.findikaattori.fi/

Case study: Better results more value -project

 45

The starting points of the project

Research and Innovation Council, chaired by the Prime Minister, proposed at 2010 for Tekes
(Finnish Funding Agency for Technology and Innovation) and the Academy of Finland a joint
task in order to develop a comprehensive framework for measurement the impacts of R&D
until the end of 2011. So far the result of this large co-operative development is a report
called Better results more value, published in December 2011.

As the starting point of the project, the report describes the “state of the art” in the
understanding and the knowledge of the R&D as follows:

“Public research and development (R&D) spending has been shown to have evident impacts
on the R&D intensity of the companies and their innovation capital as well as renewal and
productivity of the companies. However research and innovation have also various different
environmental, cultural, and societal impacts. These impacts have been studied in fewer
details than the economic ones. However, great societal challenges get more and more
focus both in innovation policies and in the strategies of innovation intensive companies.
Many of the impacts are aimed and expected, but unwanted and unexpected impacts also
occur. The scope, content and timeline of the impacts vary a lot. The interest in assessing
the impact of research and innovation has been continuously increasing due to the need to
understand the role of innovation in the competitiveness and renewal of the economies.

In addition, the research has been motivated by the need to legitimate public spending on
R&D; evidence on the impacts based on systematic research and relevant indicators have
been asked for. Another driver is the increasing need to link research and innovation policy
measures with the broader objectives in the society. This is directly evident in the EU level
where research and innovating policy development is tightly linked to the socio-economic
objectives of the EU 2020 strategy, broader societal impact of public research and innovation
funding and especially the so called societal grand challenges. Much less attention has been
laid on the role of research and innovation in other areas of the society”.

The proposed indicators of the framework can be seen in figures 4–7, as described in our
Riga 2012 working group presentation:

Figure 4. Indicators related to economy and economic renewal

Phenomenom Indicators

National prosperity GDP per capita

Overall productivity of the Total Factor Productivity TFP

economy Productivity renewal indicator

Foreign Direct Investments Share of Foreign Direct Investments per GDP

Strengthening of intangible assets Share of new innovative products and services from
business turnover.
Volume and share of intangible investments
Position in global value-networks Exports of knowledge-intensive sectors

Continuous improvement of Development of turnover in knowledge intensive sectors (or

competitiveness alternatively in KI jobs)

Collaboration, networks and
knowledge flows
Capability to innovate
Investments in R&D&I
Human resources for R&D&I
General conditions and incentives
for R&D&I
Figure 5. Indicators related to environment
Phenomenom
State of the Finnish environment
Climate change
Biodiversity
Sustainable consumption of
natural resources
New information and knowledge
related to environment
Environmentally bening
innovations
Green business
Consumers' attitudes and
behavior
R&D&I activities directed towards
environment (quality, challenges,
extent)
Cooperation within environment
related value networks and
strengthening of skill flows
R&D&I investments on
environment
Skills and human resources
Operational environment
supporting environmentally
benign actions
46
Share of public and private organisations having
collaborated in innovation projects
Development of patenting, registered trademarks and
designs (EPO / USPTO / TRIAD)
Share of R&D&I expenditure in business turnover
Government direct & indirect support to business R&D
Foreign direct investments in Finnish R&D&I
Availability of highly educated workforce
GDP share of VC investments at different growth stages
Indicators
Water systems ecological state
Green house gas emissions in Finland
Endangeredness of Finnish species
Share of renewable energy in energy production
Scientific environmental publications among the top 10%
most cited publications worldwide as % of total scientific
publications of the country
Use of environmental information in political decision making
International patents/forwarded references of patents in the
environmental sector
Revenue of companies in energy and environment sector
Private sector energy efficiency
Household energy efficiency
Companies with innovation operations in the energy and
environmental sectors
Amount of environmental R&D&I activities in research
institutions
Cooperation between private sector, universities and
research centers in environment sector
Private R&D&I expenditure on environment
Public R&D expenditure on environment
Venture capital directed towards environment
No relevant indicator so far
No relevant indicator so far
 47

Figure 6. Indicators related to well-being

Phenomenom Indicators

Health and quality of life Life expectancy

Well-being in working life Share of 25 to 64-year-olds very or fairly satisfied with their
current job
Healthy and safe living No indicator selected yet
environment
New knowledge and competence Scientific publications among the top 10% most cited
associated with well-being publications worldwide as % of total scientific publications of
the country
Innovations and systemic New products, processes, services and social innovations
changes supporting well-being
High-quality and innovative well Productivity of the social and health services of municipalities
being services and federations of municipalities
Quality and extent of R&D&I Share of public organisations involved in health and well-
activities directed towards well- being related R&D&I activities
being
Interorganisational collaboration Mobility of researchers in the fields of health and well-being
related to well-being in value
networks and the strengthening
of flows of know-how
R&D&I investments on well being Private and public R&D expenditure on well-being, health care
and working life
Knowledge and human resources No indicator at present

Supportive operational Health and social care costs

environment
 48

Figure 7. Indicators related to knowledge, education and culture

Phenomenom Indicators

Competences and opportunities Education level of population

for life-long learning

Education and active citizenship Interest in science, research and technology

Active and diverse cultural life
Openness, diversity and
networks
The quality and efficiency of the
educational system
The quality and efficiency of
higher education and research
Knowledge as a resource for the
economy and society
Citizen participation
Value added in the cultural sector
Share of foreign nationals in the human resources of science
and technology
OECD international student assessment - PISA
Scientific publications within 10% of the most cited publications
worldwide as % of total scientific publications of the country
Patent applications by institutes of higher education and public
R&D-institutes
Participation of population aged 18+ in lifelong learning

Active and vital cultural life No indicator selected at present

Internationalisation and No indicator selected at present

openness in research activities

Scientific research and education Share of doctors of the Human Resources in Science and
Technology

Disseminating research No indicator selected at present

information to citizens and the
use of society

Research and innovation No indicator selected at present

activities related to culture

International mobility and Researcher mobility (inwards and outwards)

cooperation in research
Investments in competences and Investments in R&D activities in the public sector and in the
human resources higher education sector

Investments in competences and Research personnel's share of workforce

human resources

Investments in general education Costs from adult education

and adult education
 49

Investments in the culture related The Government R&D funding based on societal objective:
to research and innovation culture
Investments in international The share R&D expenditure from abroad in the Higher
cooperation and networking Education and Government sectors

A short assessment of the project

Any deeper analysis of this framework goes beyond our possibilities and the target of this
short paper. However in the following we are contemplating the possibilities of utilising this
framework in Finnish R&D policy. This has at least two important conditions, namely:
1. Clear links to general policy and separate policies in the field of R&D have to
be established
2. Careful and well-resourced implementation has to be ensured.

The first condition for a successful implementation may be clarified with ESED sourcebook
and its definition of indicators which are as follows:

“An indicator is a quantitative measure of a phenomenon whose evolution, under given

conditions, can be seen as a consequence of a policy aimed at solving a given problem
(even if it does not represent the whole problem, but only one of its aspects). Under this
definition, an indicator (or a series of indicators) helps in evaluating the performance of a
policy. Indicators may refer to a policys inputs (the resources devoted to the policy), outputs
(a measure of the intervention carried out, e.g., the number of firms assisted), outcome
(results) (the short term changes in the enterprise system brought by the intervention, e.g.
the investments carried out by those firms) or impacts (the medium-long term changes
brought about by a policy, e.g., a permanent increase in their unit value added). Intervention-
related indicators may be usefully read against context indicators (showing the level or trend
of the same variable in the assisted area). The same indicators can also show the extent to
which an intervention has reached some target groups (e.g, number of investment projects
carried out by women).”21

The Finnish R&D indicator framework proposed by the TEKES project helps us to measure a
lot of things inside, outside, behind, in front of and besides research and development for the
needs of R&D governance. But what are the specific “needs”? What is the relation between
policies/policy and the framework of indicators? The proposed framework of R&DI -indicators
is not (pre)approved, validated and legitimated by politicians. This means that the report
Better results more value includes significant and formidable amount of work by researchers,
development personal of R&D organisations etc., but it does not include any input from the
policymakers themselves.

In this situation, the major difficulty of outcomes evaluation is not determining whether there
are effects, or the breadth or depth of them, but rather determining their merit, worth, and
significance22 in relation to politics and policy. Compared with the Findicator effort by the
Prime Minister´s Office of Finland, the main difference is that in the Findicator framework,
policies form the backbone for the indicators but this is not the case in the Better results
more value project. Policies should be located in front of the project but they seem to be far
behind it.

21
http://ec.europa.eu/regional_policy/en/
22
See Coryn 2007, 53-54.
 50

This is not only a Finnish problem. As the White Paper draft of KNI describes it, “One of the
problems hindering the implementation of these tasks is that the information needs of
decision-makers are not systematically met and analysed. Because of this the auditors need
to make special efforts to overcome these difficulties”. This means that in every INTOSAI
country, auditors have to ask the same question which the director of the Research and
Innovation Council of Finland asked us in a discussion last January: “How many busy
politicians are able or willing to stop for a while and study this large and ambitious report?”

This problem extends far outside the R&D-field. The OECD Report of Governance in Finland
reports that our problems in general state governance are deeper than in the “technical
formulation” of indicators for effectiveness and performance of authorities. The OECD report
tells us that:

“The state performance-management system seems to fall short of supporting an overall

strategic focus that connects agency performance objectives to the achievement of societal
outcomes. Ministry and agency officials report that performance measures tend to focus on
detailed processes rather than strategic actions that support the Government Programme. At
the organisational level, there seem to be few, if any, consequences for failure to meet
strategic objectives. This shortfall appears to stem from four main factors:

1) lack of clear, strategic whole-of-society vision communicated by government;

2) difficulty in developing indicators that clearly link back to strategic whole-of-society
objectives;
3) insufficient prioritisation of overall objectives; and
4) lack of real accountability, in particular for shared strategic objectives.

Ministries struggle to develop indicators to achieve unclear outcomes, and performance

discussions with agencies seem to focus on process indicators rather than contributions to
overall objectives, demonstrating greater comfort with those indicators that are most easily
measurable”.

Tekes and the Academy of Finland are required to report about the implementation of the
project in November 2012. The NAO of Finland intends to follow the project as part of our
routine risk analysis.

6. The role of SAI of Finland in the developing the main indicators of R&D

So far Finland has kept an outsider´s view to Finnish indicator system of R&D. As the White
Paper on Key National Indicators puts it, “the process of selecting key national indicators is
inherently political, representing the aspirations and values of society”. In Finland we are just
now starting and strengthening this process, or path, both at the political level and at the
methodological level.

In order to start this path, the SAI of Finland has organized two workshops in the field of R&D
and innovation last year. On March 2012 we also organized a brainstorming session about
the challenges in the governance of the Finnish education, research and innovation system.
In addition to our workshops and brainstorming sessions we have done some important
audits in this field. The performance audit of Finnish R&D evaluations (in 2008) and the audit
of the governance of human resources of Finnish universities (2010) are two our major
efforts. Just now we are starting a new audit concerning the governance of the Finnish
education, research and innovation system.

The most important challenge at the SAI of Finland is to reflect our position and find the best
way to function with the national key indicators of Finnish R&D rather than to take a role of
 51

advisor and expert. Of course this does not mean that we should be passive and just wait
what is going on. The SAI of Finland has been very active in promoting discussion about the
state and quality of Finnish innovation system and the evaluation of the system. As the KNI
white paper draft formulates it, “It is necessary to combine the efforts of all experts, dealing
with the issues related to the assessment of the effectiveness of socio-economic
development strategies implementation, as well as to bring on board other interested
parties”.

7. Conclusions

Our main conclusion could be formulated as follows: in the near future the approach to R&D
indicators involves working together, collaborating and reflecting rather than being busy
measuring them. As far as we are able to see, this means that indicators are not and they
should not be direct guides or maps to decisions about the ends and means of R&D in
Finland or in some other country.

Four general roles of indicators seem to us to be most important if we are going to encounter
the complexity of the R&D and innovations with our indicators successfully:
 we have to see indicators as a collective enterprise to catch up the complexity
of human needs (cultural, social, environmental, economic etc.) beyond R&D
and innovations, rather than an evident or dominant yardstick for setting social
goals to society or the R&D actors,
 we have to see indicators as a diagnostic of possibilities of R&D and
promising ways to enlarge them rather than a problem solver in the details
(even though the devil often seems to be in the details),
 we have to see indicators as combining needs and possibilities in a new,
productive, and innovative ways rather than making definitive tests or
confirmation of old or new ideas and solutions,
 we have to see indicators as a candle of the wholeness of R&D and society
rather than a bright light bulb for the parts of the whole (economy,
environment etc.).

This means that the development of R&D indicators in Finland is going in right direction.
However, these conclusions do not mean to say that the analysis and evaluation of the
so- called KES (Knowledge-based economy and society) should be ignored.

The role of SAO in this project is not easy to define. So far the Finnish starting point,
as presented at the working group’s Riga meeting on April 2012, is shortly described as
follows:
 52

Figure 8. Indicator development and the SAI’s role

As the OECD project shows, a couple of years ago there were still many barriers in front of
the knowledge-based economy and society (ideological, practical etc.)23. However, a special
challenge for SAIs and their role at the development of indicators as a part of the governance
of the whole is the paradigmatic change towards s new socio-political regime. This change
involves radically new kind of challenges and risks for SAIs in every country. We finish our
paper by asking, what would and could be the role of SAIs and Key National Indicators of
R&D and education in this new future of governance: global, regional, national, or local?
Without this kind of reflection we are not able to see ourselves as SAIs in a valid mirror.

23

http://www.oecd.org/sti/scienceandtechnologypolicy/governanceofinnovationsystemsvol1synthesisrep
ort.htm
 53

REFERENCES:

Coryn, Chris (2007): Evaluation of Researchers and their Research: Toward making the
Implicit Explicit. Western Michigan University, Kalamazoo, Michigan 2007.

European Union (2003): Evaluating Socio Economic Development. Sourcebook 1: Themes

and Policy Areas Enterprise policy.

European Union (2011): Monitoring Policy and Research Activities on Science in Society in
Europe (MASIS) National Report, Finland October 2011.

Lemola, Tarmo & Kanninen, Sami (2006): Methods for Evaluating the Impact of Basic
Research Funding. Publications of the Academy of Finland.

OECD (2006): Governance of Innovation Systems, Vol. 1: Synthesis Report.

Patton, Michael Quinn (2011): Developmental Evaluation. Applying Complexity Concepts to

Enhance Innovation and Use.

Ritsilä Jari, Nieminen Mika & Sotarauta Markku (2007): Societal and economic engagement
in universities. An evaluation model and views on the roles of universities. Reports of the
Ministry of Education, Finland 22/2007.
http://www.minedu.fi/OPM/Julkaisut/2007/Yliopistojen_yhteiskunnallinen_vuorovaikutus.html
?lang=fi&extra_locale=en

Tekes – Finnish Funding Agency for Technology and Innovation (2011): Better results, more
value. A Framework for analysing the societal Impact of Research and Innovation. Tekesin
katsaus 288/2011.
 54

Annex E

Analysis of opportunities on the use of KNI for international

comparisons in the context of sustainable development and
recommendations

A/ Background and objectives of this analysis

Prior to the working group meeting held in Riga in April this year, we addressed two
questions to the members of the working group:
1. Whether your SAI participates in the elaboration or development of the national indicator
system of sustainable development (and if so, please indicate how) and
2. Whether your SAI has already tried to utilise these indicators during SAI audits, especially
performance audits (if so, please indicate how and in which topics).
The questions were accompanied by a description of the Hungarian situation on the above
subject.
We received eight responses to the questions, and the working group meeting referred to
above provided further valuable input on the issue.
In the meantime, we used examples from specific audits to review the applicability of
sustainable development indicators (referred to in this paper as key indicators or KNI) in the
practice of the State Audit Office of Hungary.
Section B of this discussion paper contains an analysis of the issue indicated in the title,
providing a summary of the above inputs as well as a brief description of INTOSAI’s relevant
principles and the EU’s system of sustainable development indicators as well as other
indicator alternatives. That is followed, in Section C by our recommendation, which, based on
our analysis of the inputs, we hope to implement through the intensive involvement of
working group members.
 55

B/ Analysis of the application of key indicators by supreme audit institutions

Ba) The role of SAIs in promoting sustainable development

INTOSAI’s ISSAI 5130, Sustainable Development: The Role of Supreme Audit Institutions,
explained below in brief, serves as the basis and standard for the objective referred to in the
title.24
The now classic definition of sustainable development, first offered by the 1987 Brundtland
report (“development that meets the needs of the present without compromising the ability of
future generations to meet their own needs”) continues to apply. Additionally, especially from
the perspective of SAI audits, the balance and harmony of social, economic and natural
environmental objectives are also important considerations. Governments play a key role in
the application of sustainability criteria, which is thus also relevant to SAIs (especially where
the mandate of the SAI includes the preliminary assessment of budget appropriation bills).
As SAIs are typically auditors of implementation rather than policy setters, their role in the
promotion of sustainable development is delimited by the extent to which the government
concerned has adopted the principles and practice of sustainability.
Naturally, the specific way in which an SAI contributes will depend on the mandate of that
SAI. Despite the differences, there are general common requirements for government
strategies for sustainable development and the implementation of such strategies, the control
of which carried out by auditors is assisted by the key audit issues listed in the ISSAI
document referred to above. The right selection of indicators to measure sustainability
performance is of key importance. Sustainability audits are concerned with the objectives set,
the indicators measuring adherence to those objectives and progress made in adherence, as
well as whether the institutional conditions are provided for setting and adhering to
sustainability targets.
In the absence of a sustainability policy at the level of government, auditors can review
individual programmes in terms of sustainability. In such cases, selecting the right
programmes for review is of key importance, and there is also a methodological challenge in
examining the connections and trade-offs between competing economic, social and
environmental objectives. The document referred to provides specific examples of this.
An SAI can control sustainable development only if it builds suitable institutional capabilities,
which will involve its mandate and strategy, the establishment of competent auditor
capacities and the specification of audit procedures, as well as the follow-up of auditors’
findings and recommendations.

24
The document is available at www.issai.org under Auditing guidelines on specific issues.
 56

Bb) Summary of Hungarian audit experience gained after the working group
meeting in Riga
In the spirit of the ISSAI document referred to above, we have addressed whether and how
key sustainability indicators (KNI) can be applied by the State Audit Office of Hungary, as
part of which we have recently reviewed three audit programmes: those of public
employment and related training, hospitals, and national parks.

From our review of the programmes, we have drawn the following conclusions, which may
also be utilised in our working group:

 It is an important prerequisite to provide the conditions for accountability in good time

prior to an audit, since the SAI obviously cannot impose requirements for
performance that are entirely new and unfamiliar to the institution or sector under
review. Where it exists, a sectoral strategy can serve as the basis for accountability,
provided that it includes key indicators which the SAI finds relevant. In the absence of
such a strategy, the SAI can make a proposal for the elaboration of a target
programme to govern the institution or sector, including setting objectives that can be
measured against generally accepted sustainability indicators.

 A general methodological problem involved in the SAI’s application of key indicators

is to bridge the level difference between macro indicators and audits carried out at the
level of sectors or institutions. The problem can be resolved more easily where it is
possible to interpret and calculate the indicator at the level of sectors and institutions.
The task is more complex, for example, in the case of healthcare, where
effectiveness indicators depend on a large number of factors, among which the
healthcare system is important, but far from being the only factor.

A pivotal issue of performance audits, including the application of key indicators, is

the appropriate definition, delimitation and weighting of criteria for results, interim
results and efficiency within the programme. Efforts should be made to avoid
confusion of these basic indicator types, and overlaps between target and tool
indicators and between effectiveness and efficiency. In terms of the healthcare
example, indicators of access to a service (geographical availability, waiting list) or
the patient throughput indicators of a healthcare institution are important efficiency
measures, which, however, cannot replace the actual result of the impact on the
health of the population.

 The internal regulation of the SAI should give weight to the indicators selected for
performance audits (such as the sustainability indicators of the national statistical
office or an international organization), and the application of indicators should be
 57

incorporated into the methodological regulation of performance audits. Failure to do

so could displace the consideration of sustainable development in the SAI’s multi-
factor and multi-player procedure for audits.

Bc) Thematic basis of international comparison

Our analysis draws on the experience of eleven countries including Hungary.

As far as keynote question 1 invoked in Section A is concerned, in the vast majority of
respondent countries SAI participation in the elaboration of sustainable development
indicators is non-existent or marginal.
In respect of question 2 concerning the SAI’s utilisation and the relevance of key indicators,
practice varies considerably across the eleven countries.
Two countries reported that their SAIs were not using key indicators (KNI) in their work (or
not in the form indicated in the question).
At present, key indicators play a special role in the practices of three countries:
 Denmark’s presentation at the working group meeting addressed the audit of key
indicators themselves, which is in line with the fact that Denmark is one of the
countries the SAIs of which do not participate in the elaboration of key indicators.
 In response to question 2, Russia reported to have applied key indicators in country-
specific audits of larger regions and sectors (e.g. the oil industry).
 In Riga, Austria presented an interesting aspect of the application of key indicators,
namely the fiscal sustainability of the country in the long-term, with special regard to
the effects of its ageing population on the pension system, and on the costs of
healthcare and social care. A comparative analysis of the indicators of overall fiscal
sustainability should be attempted at a later stage of our key indicator project when
more audit experience is available; however, the “ramifications” in terms of the
pension system, healthcare and social care addressed in the Austrian presentation
could be suitable themes for an internationally coordinated methodological analysis
even today.

The SAIs of the rest of the countries also reported themes of key indicator application that
may be of interest to others. These themes are the following (frequencies are indicated in
brackets including Hungarian audit themes and the Austrian subthemes referred to above):
 Environment and environmental management (4);
 Healthcare (3);
 Social care (2);
 58

 Public education, public employment and related training (2);

 Research and development (1).

The international sample represented by the working group suggests that SAIs most typically
apply key indicators in themes relating to environmental management. Human resources
management is represented by three themes (mentioned by 2 or 3 countries each).
Research and development, presented by the Finnish SAI, is a unique theme of indicator
application, which, at the same time, is relevant to sustainable economic development.
We think that the above thematic distribution of key indicator applications could be used to
develop a common thematic denominator among working group members, which could serve
as the basis for a pilot project on the comparison of key indicators.
Thematic coordination is required but not sufficient for a comparative project on key
indicators. The other prerequisite is agreement on the indicators to be applied, mindful of
international comparability as set forth in Subsection Bd and Be:

Bd) Laying the foundations of international comparison using Eurostat indicators

In terms of indicators, the indicators underlying the strategy of the European Union on
sustainable development could be an obvious basis for international comparison. Eurostat
regularly publishes data using those indicators (most recently in 2011). The upper level of the
hierarchical indicator system comprises 11 headline indicators representing ten challenges.
More than a hundred additional indicators explain and detail the challenges hallmarked by
the headline indicators (refer to the Annex 1 for the headline indicators and subthemes of the
Eurostat sustainability indicator system).
The classification of Eurostat indicators around challenges such as social inclusion, climate
change and energy, etc. could be of benefit in laying the conceptual foundations of policies to
address the challenges concerned; however, it could pose difficulties in the case of a
practical task such as the application of key indicators by SAIs. For instance, indicators
relating to environmental protection are classified under at least four different headline
indicators.
Below is an overview of the themes audited and analysed by the working group in terms of
whether they can be adequately supported by current Eurostat indicators:

Scattered around a number of challenges, there are plenty of indicators addressing the
theme of environmental protection and natural resources management:
 In Theme 1, Socioeconomic development, under energy intensity (energy requirement
of the economy relative to GDP);
 59

 In Theme 2, Sustainable consumption and production, subtheme Resource use and

waste comprises as many as ten indicators describing material consumption and
waste management, including four emission-type indicators;
 Theme 6, Climate change and energy, is comprised of a total of 14 (8+6) indicators;
 Theme 7, Sustainable transport, includes one indicator relating to energy consumption
and four emission-type indicators;
 Theme 8, Natural resources, comprises eight indicators of relevance, concerning in
part exploitation and in part status.

With the sustainability of human resources, “indicator coverage” presents a more varied
picture:
 Theme 5, Public health, includes a relatively small number of indicators supporting
healthcare, comprising four status indicators and four health risk indicators.
 Scattered across themes, a total of ten indicators describe employment: six
employment indicators in Theme 1, Socioeconomic development, three in Theme 2,
Social inclusion, and, oddly enough, the headline indicator of Theme 4, Demographic
changes, is also essentially an employment indicator.
 Divided into two groups, another ten indicators describe poverty and inequality, eight
in Theme 2, Social inclusion, and two in Theme 4, Demographic changes (note that
Theme 4, Demographic changes, includes at least six indicators that could support an
analysis of the effect of the ageing population addressed in the Austrian
presentation).
 Finally, in Theme 2, Social inclusion, five indicators are concerned with training and
qualifications, mostly at an elementary level and from the perspective of social
integration.

Theme 1, Socioeconomic development, includes only one Eurostat indicator to support

the analysis of research and development.

The above overview of Eurostat indicators suggests that the fields of indicator application as
reported by working group members have a rather uneven coverage of Eurostat indicators.
The largest number of Eurostat indicators support the themes of environmental protection
and environmental management, followed by social policy, employment, health and training,
with research and development at the end of the line.

Be) Other indicator alternatives

 60

Taking into account the very high complexity of dimensional interactions of Sustainable
Development a vast range of indicators is in need for having a conceptual framework. It has
a key role in organising and relating information on economic and environmental and social
development, as well as on interactions of current flows and their impact on long-term
development. The desired characteristics of conceptual frameworks for Sustainable
Development indicators might be listed as follows:

 to present a way of integrating economic, environmental and social

dimensions of Sustainable Development,
 providing approach as a basis for selecting indicators being analytically sound,
policy relevant and measurable,
 to be transparent and simply to understand,
 to cover information over long period of time and data to interpret results
(benchmarks).
Of course, these characteristics not individually but all together could meet the requirements
of a conceptual framework.

The SEEA25 handbook (jointly published by the United Nations, the OECD, the International
Monetary Fund, the European Commission and the World Bank) contains an overview of
various ways to put into operation the original definition of Sustainable Development
proposed by the Brundtland Commission. In particular, three main approaches or conceptual
frameworks are identified26:

 the three-pillar approach;

 the ecological approach;
 the capital approach.
The first approach (framework) must contain no single focus (or object) of sustainability, but
instead all economic, social and environmental systems must be simultaneously sustainable
in and of themselves. This framework is also known as theme-based approach covering
social, environmental, economic and other aspects, e.g., institutional aspect of Sustainable
Development. The indicator systems of the United Nations (see Annex 2), the OECD (see
Annex 3) and the European Union (Eurostat) are based on such an approach (framework).

25
System of Environmental-Economic Accounts. The SEEA Central Framework was adopted by the
UN Statistical Commission in 2012 as the first international standard for environmental-economic
accounting.
26
United Nations, European Commission, International Monetary Fund, Organisation for Economic
Co-operation and Development, World Bank, 2003. Integrated Environmental and Economic
Accounting 2003. Final draft.
Available at the UNSD website: http://unstats.un.org/unsd/envaccounting/seea2003.pdf
 61

Central to the ecological approach (framework) of Sustainable Development is the notion that
economic and social systems are sub-systems of the global environment. Therefore, it
follows that sustainability in the economic and social spheres is subordinate to sustainability
of the global environment. Due to its one-sidedness it was used rather in the early years of
Sustainable Development.

Finally, the capital approach (framework) borrows the concept of capital from economics, but
broadens it in a variety of ways to incorporate more of the elements that are relevant to the
sustainability of human development. In doing so, it takes concepts from the physical
sciences (especially ecology and geography) and from the non-economic social sciences
and integrates them within a framework based on capital. This framework worked out by the
Joint UNECE/OECD/Eurostat Working Group on Statistics for Sustainable Development.27

The underlining idea is that an indicator of total national wealth is, in theory, an ideal indicator
of Sustainable Development.

The challenge of Sustainable Development has been simplified into a question of whether a
country’s total capital base – or total national wealth – is managed in a way that secures its
maintenance over time. Thus simplified, the focus of the Sustainable Development challenge
is sharpened and put into concrete terms. The question whether financial, produced, natural,
human and social capital stocks per capita are increasing or declining over time is one that
lends itself to a precise answer. Furthermore, this focus helps make sense of the inevitable
tradeoffs that must be weighed as development proceeds. For example, if one capital stock –
let us say, petroleum wealth – declines, the framework allows us to ask whether it is being
offset by growth of another stock, human capital perhaps. This last question touches on a
difficult point of whether, and to what extent, the various capital stocks can be expected to
substitute for each other as far as well being effects are concerned.

As can be seen, the proposed small set of capital based indicators in Annex 4 has been
divided into two indicator domains. The first is labelled foundational well-being to reflect the
fact that the indicators measure stocks and flows that are essential to the well-being of
society. The second domain is labelled economic well-being. The indicators within it are
more narrowly related to the well-being derived from market activity.

There are 28 indicators in the proposed small set. The indicators in the small set represent a
theoretically robust, substantially complete and policy-relevant approach to measuring

27
United Nations, European Commission, International Monetary Fund, Organisation for Economic
Co-operation and Development, World Bank (2003): Integrated environmental and economic
accounting 2003, Studies in Methods, Handbook on National Accounting, Series F, No. 61, Rev. 1,
(ST/ESA/STAT/SER.F/61/Rev.1) SEEA 2003.
(http://unstats.un.org/unsd/envaccounting/seea2003.pdf)
 62

Sustainable Development. Any country that compiled them all would be in a very good
position to report upon its potential for sustaining well-being in the long term. If many
countries were to compile them as part (or all) of their national Sustainable Development
indicator sets, the basis for comparing progress across nations in terms of achieving
sustainable development would be greatly improved.

The set is not of as much use for reporting on the elements of current well-being, though it is
far from useless for this purpose. The set also cannot correspond perfectly to the policy
priorities in all countries. For both these reasons, any given country might feel that the
proposed small set is insufficient to meet its needs for measuring Sustainable Development.
To the extent that this is true, the small set can, of course, be supplemented with additional
indicators reflecting the national situation.

C/ Recommendation

In the spirit of INTOSAI’s ISSAI 5130, Sustainable Development: The Role of Supreme Audit
Institutions, and with a view to its adoption in auditors’ practice, the working group should
involve its members in the initiation of a pilot project to test the audit application of key
sustainability indicators:

 based on Eurostat indicators or other sets of indicators referred to in Subsection Be

agreed upon by the working group;
 in two themes out of the audit themes referred to in Subsection Bc (environmental
protection, social care, employment, health, and possibly training).

Qualifying members of the working group should undertake to call for or enforce application
of the relevant sustainability indicators in audits launched on the above themes over the next
two years, and prepare an international comparative analysis and recommendation on their
findings. Working group members whose national institutional circumstances prevent them
from undertaking this task could contribute to the success of the proposed key indicator
project in the capacity of advisers.

October 2012
 63

ANNEX 1
Sustainable Development Indicators in the European Union
(1st level and subthemes)

(Source:
Sustainable development in the European Union
2011 monitoring report of the EU sustainable development strategy)

1. Socioeconomic development
Headline indicator: Real GDP per capita
Subtheme: Economic development
Subtheme: Innovation, competitiveness and eco-efficiency
Subtheme: Employment

2. Sustainable consumption and production

Headline indicator: Resource productivity
Subtheme: Resource use and waste
Subtheme: Consumption patterns
Subtheme: Production patterns

3. Social inclusion
Headline indicator: Risk of poverty or social exclusion
Subtheme: Monetary poverty and living conditions
Subtheme: Access to labour market
Subtheme: Education

4. Demographic changes
Headline indicator: Employment rate of older workers
Subtheme: Demography
Subtheme: Old-age income adequacy
Subtheme: Public finance sustainability

5. Public health
Headline indicator: Life expectancy and healthy life years
Subtheme: Health and health inequalities
 64

Subtheme: Determinants of health

6. Climate change and energy

Headline indicator: Greenhouse gas emissions
Headline indicator: Consumption of renewables
Subtheme: Climate change
Subtheme: Energy

7. Sustainable transport
Headline indicator: Energy consumption of transport relative to GDP
Subtheme: Transport and mobility
Subtheme: Transport impacts

8. Natural resources
Headline indicator: Abundance of common birds
Headline indicator: Conservation of fish stocks
Subtheme: Biodiversity
Subtheme: Freshwater resources
Subtheme: Marine ecosystems
Subtheme: Land use

9. Global partnership
Headline indicator: Official development assistance
Subtheme: Globalisation of trade
Subtheme: Financing for sustainable development
Subtheme: Global resource management

10. Good governance

Subtheme: Policy coherence and effectiveness
Subtheme: Openness and participation
Subtheme: Economic instruments
 65

Annex 2

Selection of UNCSD Core Indicators by Testing Countries

New Indicators Suggested by

Indicators Frequently Used
Countries

Unemployment Rate Incidence of environmentally

related disease

Population growth rate % Population with access to

health services

GDP per capita Crime rate

Domestic per capita consumption Incidence of street children

of water

Land use change Urban green space

Use of fertilizers Ground water pollution

Ratio of threatened species to Ratio of mining area

total native species rehabilitated to total mining area

Ambient concentration of urban air Area of specific ecosystems

pollutants

Emissions of greenhouse gases Ownership of agricultural land

Emissions of sulphur dioxides Genuine savings ratio

Emissions of nitrogen dioxides Traffic density

Annual energy consumption Release of GMOs

Source: UN, United Nations. 2001. Indicators of Sustainable Development: Guidelines and
Methodologies. New York.

(http://www.un.org/esa/sustdev/publications/indisd-mg2001.pdf)
 66

Annex 3

OECD core set of Sustainable Development indicators

Source: OECD Sustainable Development: Critical issues. Paris, 2001.

 67

Annex 4

A proposed small set of sustainable development capital based indicators

Indicator domain Stock Indicators Flow Indicators

Foundational well-being Health-adjusted life Changes in age-specific mortality

expectancy and morbidity (place holder)
Percentage of population Enrolment in post-secondary
with post-secondary education
education
Temperature deviations from Greenhouse gas emissions
norms
Ground-level ozone and fine Smog-forming pollutant
particulate concentrations emissions
Quality-adjusted water Nutrient loadings to water bodies
availability
Fragmentation of natural Conversion of natural habitats to
habitats other uses
Economic well-being Real per capita net foreign Real per capita investment in
financial asset holdings foreign financial assets
Real per capita produced Real per capita net investment in
capital produced capital
Real per capita human Real per capita net investment in
capital human capital
Real per capita natural Real per capita net depletion of
capital natural capital
Reserves of energy Depletion of energy resources
resources
Reserves of mineral Depletion of mineral resources
resources
Timber resource stocks Depletion of timber resources
Marine resource stocks Depletion of marine resources