See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/323000748

Smart city performance measurement framework CITYkeys

Conference Paper · June 2017

DOI: 10.1109/ICE.2017.8279956

CITATIONS READS

2 299

6 authors, including:

Aapo Huovila Hans-Martin Neumann

VTT Technical Research Centre of Finland AIT Austrian Institute of Technology
28 PUBLICATIONS   149 CITATIONS    17 PUBLICATIONS   32 CITATIONS   

SEE PROFILE SEE PROFILE

Peter Bosch
TNO
30 PUBLICATIONS   3,784 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Resilient Cities and Infrastructures (RESIN - Horizon2020) View project

SMARTER TOGETHER View project

All content following this page was uploaded by Aapo Huovila on 09 February 2018.

The user has requested enhancement of the downloaded file.

 Smart City performance measurement framework
CITYkeys
Miimu Airaksinen, Isabel Pinto Seppä, Aapo Huovila
VTT Technical research centre of Finland
Espoo, Finland
isabel.pinto-seppa@vtt.fi
Hans-Martin Neumann, Branislav Iglar
AIT Austrian Institute of Technology
Wien, Austria
Abstract - This paper presents a holistic performance
measurement framework for harmonized and transparent
monitoring and comparability of the European cities activities
during the implementation of Smart City solutions. The work
methodology was based on extensive collaboration and
communication with European cities in order to develop a set of
Key Performance Indicators (KPIs) specific for Smart Cities
initiatives evaluation and comparability. The baseline was
established by analysis and integration of existing results from
previous initiatives. The research developed and validated a
performance evaluation framework, including KPIs definition,
guidelines for data collections, a performance system prototype
and testing in case-cities. In addition recommendations for the
implementation of the performance system into the cities
decision-making process and recommendations for the
development of new business opportunities are discussed.
Keywords—Smart City; perfomance assessment; Key
performance indicators; city open data.
I. INTRODUCTION (HEADING 1)
Whereas more than half of the world’s population lives in
cities, this rises to over two thirds in EU28 and the proportion
is growing. High density city populations increase strains on
energy, transportation, water, buildings and public spaces [1].
Urban areas account for 70% of current global CO2 emissions
and hence heavily contribute to the threats of global climate
change, while simultaneously being highly vulnerable to the
impacts of it. This causes extensive challenges in the urban
areas, for example regarding air pollution, congestion, waste
management and human health [2]. The creation and
implementation of action plans is crucial for the raise of
sustainable urban areas.
According to [3, 4] sustainability is based on a principle
where everything that we need for our survival and well-being
depends, either directly or indirectly, on our natural
environment. Sustainability creates and maintains the
conditions under which humans and nature can exist in
978-1-5386-0774-9/17/$31.00 ©2017 IEEE
Peter Bosch
TNO - Netherlands Organisation for Applied Scientific
Research
Delft, Netherlands
l
productive harmony, that permit fulfilling the social, economic
and other requirements of present and future generations.
Cities are areas of creativity and of economic growth: the
potential for exchanges, optimization and new solutions is
unique and enormous. However, this transition process is
progressing slow [e.g. 5, 6]. As the EU has set its climate and
energy targets for 2020 and 2030 [7] there is an urgent need to
develop smart solutions to overcome barriers and to address
these challenges [8], and accelerate transition. Innovative
approaches are needed to tackle problems related to
overcrowding and jamming of infrastructures, energy
consumption, resource management and environmental
protection. The development of Smart Cities solutions is highly
relevant in order to contribute to the 20-20-20 targets
established in the European 2020 Strategy and, beyond that to
the European 2050 objectives.
The term ‘smart city’ often implies a usage of ICT solutions
in the city [9]. Some definitions use intelligent cities and smart
cities as synonymous terms [10] while others make a
distinction [9]. The intelligent city refers to a city that has an
information technology infrastructure [11]. The ultimate
objective of smart city is to be sustainable [12].
In this context, performance analysis has become an
important tool in city development planning phases or in
project assessment, but also in assessing cities. City rankings
are popular today and they increasingly attract public attention.
The comparison of cities can attract new resident citizens and
support investors in their choice of location. In addition it can
be an important guide for the cities to judge their strengths and
weaknesses and to define their goals and strategies for future
development and better positioning in the urban system [13].
For proper comparison of cities, setting of requirements at
planning phases and evaluation of projects performance,
appropriate evaluation mechanisms and indicators are needed.
This is also true when deploying smart city strategies and
projects. An indicator can be defined as “Anything used to
measure the condition of something of interest. Indicators are
often used as variables in the modelling of changes in complex
environmental systems” [14].
718
 City administrations tend to use a diverse set of key
performance indicators (KPIs) to evaluate the success of
specific projects. These KPIs may reflect the city’s
environmental and social goals, as well as its economic
objectives [15]. Appropriate metrics need to articulate progress
towards determined strategic goals aligned with the
sustainability principles, which can then lead cities to develop
indicators to measure success against the goals in each smart
city characteristic. It has also been suggested that it is vital that
metrics measuring success are connected to the vision and
goals [16].
Currently there are many initiatives by cities, companies,
research groups, and authorities to create methodologies or
frameworks for assessment of the sustainability or the
environmental impact of a city, including also the assessment
of smart city initiatives. Recently, several standards have been
developed using life cycle thinking to determine the
environmental impact of ICT products, networks and services.
There are also a number of initiatives where cities are to report
their greenhouse gas emissions and energy usage [17].
Most often in smart city assessment frameworks the
classical people (society), planet (environment), profit
(economy) classification is used, or in some form that
classification is behind the framework [18, 19]. Typically the
assessment frameworks have indicators in the categories of
economy, people and living, and governance and services. In
addition almost all frameworks also have indicators related to
environment, and these indicators consider mainly energy
consumption, sustainability of buildings, carbon footprint,
waste generation etc. Also smart mobility, transport and
infrastructure are addressed in the systems but there the focus
is somewhat different depending on the framework. The role of
ICT as enabling technology is embedded in all main categories.
However, until now there have not been frames focusing on
scalability on replicability of smart city solutions.
This paper presents a recent developed performance
assessment framework for Smart Cities: CITYkeys1. The
initiative was funded by the European Union and developed
and validated, with the aid of cities, key performance indicators
and data collection procedures for the common and transparent
monitoring as well as the comparability of smart city solutions
across European citie. The research developed and validated a
performance evaluation framework, including KPIs definition,
guidelines for data collections, a performance system prototype
and testing in case-cities. In addition recommendations for the
implementation of the performance system into the cities
decision-making process and recommendations for the
development of new business opportunities are discussed.
II. CITYKEYS FRAMEWORK
The CITYkeys assessment method and the indicators are to
be used to evaluate the success of smart city projects and the
possibility to replicate the (successful) projects in other
contexts. The framework developed indicators for the city level
1 indicators in terms that are familiar with the decision makers.
http://www.citykeys-project.eu/
and for project level. As follows from the smart city definition,
success is determined by the transition across the entire
environmental footprint of urban areas, simultaneously
promoting economic prosperity, social aims and resilience to
climate change and other external disturbances. The framework
is based on five major themes: people, planet, prosperity,
governance and propagation.
Over the past decennia, the concept of sustainability - split
up in the triple bottom line of social sustainability (People),
environmental sustainability (Planet) and economic
sustainability (Prosperity) - has become generally accepted in
the development of indicator systems for national and regional
urban development [20]. The 3 Ps (people, planet, prosperity)
have also gained considerable ground in company reporting
[21]. Therefore these were used as pillars for the framework
development.
The extent to which smart city projects are able to have an
effect on social, environmental and economic indicators forms
the core of the evaluation. However, this is not enough to
determine the success of a smart city project. Success is also
determined by how projects have been - or will be - realised in
various contexts. The Governance of developing and
implementing urban smart city projects is a determining factor
for high scores in People, Planet and Prosperity indicators [22].
Reference [23] also note that Governance has been established
as one of the four pillars of sustainable development. Therefore
we need to include a number of indicators to evaluate the
importance of the city context (external factors) and quality of
the development and implementation process (internal factors).
Finally, the ability of individual smart city projects to be
replicated in other cities and contexts determines their ultimate
effect in achieving European goals with regard to energy and
CO2 emissions. Under the Propagation category, smart city
projects are evaluated to determine their potential for up-
scaling and the possibilities for application in other contexts.
Each of the major themes (people, planet, prosperity,
governance and propagation) encompasses several specific
policy goals. In many cases these are not all mentioned in a
smart city strategy, but may be scattered over various policy
documents in a city. For the design of the CITYkeys indicator
framework we have arranged these policy goals under the
major theme headings. For instance, under the theme People,
subthemes conforming to policy ambitions are created (see
Fig.1): increasing diversity and improving social cohesion,
increasing safety, guaranteeing good education for every
citizen, etc.
The reasons for doing so are:
- to underline the relation between policy ambitions and
the key indicators that are to be used to measure progress
towards these ambitions
- to provide the basis for comparing the indicators with
each other, whereby users or user groups may attach
weightings to policy goals (and thereby to the indicators
belonging to a subtheme).
- to ease communication on the outcome of the
719
 Cities need a performance measurement system that can be
used for the assessment of strategies, projects but also to see
what is the contribution of a certain project to the city strategy.
The implemented system should be flexible, user-friendly,
secure, compatible and comprehensive. The CITYkeys KPI
framework contains project-to-city link for many indicators
The framework developed indicators for the city level and for
project level and contains project-to-city link for many
indicators. These were tested allowing assessing, to some
extent, the impact of a project on city level.
Fig. 1: The CITYkeys indicator framework
III. AVAILABILITY AND RELIABILITY OF KEY DATA SETS
The calculation of several indicators within the framework
demands for the availability and reliability of the needed data.
There is a difference in the data collection process between the
project and the city indicators. The project indicators are meant
for assessing the success and potential for replication of smart
city projects. For this a mixture of quantitative and semi-
quantitative indicators is proposed. For the semi-quantitative
indicators data needs to be collected using interviews or an
analysis of policy documents. For the quantitative indicators
data needs to be extracted from the project documentation.
Cities might want to streamline the data collection for all their
smart city projects, but a number of the project indicators will
require qualitative information that can only be gathered by
involving persons involved in the project (e.g. through
interviews, questionnaires).
Data for the majority of the city indicators can be retrieved
from statistical sources within the city administration or
national or European statistics bureaus. One problem with that
data is that it typically presents average annual figures for a
whole city. For a city it may however be more interesting to
analyse the differences between districts. Spatial data makes it
possible to calculate indicators also for geographically
restricted areas such as city districts.
The data sets relevant for the CITYkeys KPIs were
identified and analysed regarding their availability, sources,
reliability, data access methods, existing data formats and level
of confidentiality. Special attention was paid in the analysis of
open data sets. The data availability rates for the data sets
needed by the smart city KPIs was analysed in the for the
CITYkeys partner cities. These were, in January 2016, as
follows: Zaragoza 82%, Vienna 80%, Tampere 77%,
Rotterdam 71% and Zagreb 52%. The average availability is
72%. The share of open data providing the needed data sets
varies from 1% to 25%, and is 15% on average.
In total the smart city KPIs, in the framework, need 116
raw data sets. Depending on the city and calculation method
less than 20 of open data sets were directly applicable for the
calculation of the selected CITYkeys KPIs. This means that
roughly 85% of needed data sets are either from city’s internal
systems; are not available and need to be generated; or they are
public but do not qualify as open data. For an average of 13%
of the data sets, their availability remains unknown at this stage
of the project giving a possibility for slight changes in the
availability rates. External organizations and companies are the
most common source for needed available data sets (32%)
because some cities have outsourced their functions such as
water management, electricity. Most of the available data sets
coming from city departments (23%) originate from Mayor’s
office/Economy (including cities’ statistics departments).
On the project level, most of the project KPIs are
qualitative or semi-quantitative and need to be evaluated for
example based on project documentation or interviews with
project manager. Moreover, the boundaries of these data sets
need to be further defined in each project separately because
the coverage of relevant data is always project specific.
It is important to note that data for all indicators will
obviously not be available immediately in all cities. A city that
engages in smart city indicators starts a process. The CITYkeys
indicator framework is a methodology for such a process. The
cities will need continuous development of the indicators to be
used by the city and of the data collection mechanisms.
Although not mandatory, it could be recommended as a
CITYkeys goal that cities automate their data collection and
framework feeding.
Moreover, the definitions behind certain data sets and data
quality obviously differ within countries, between cities and
between city departments. The quality of the overall
assessment depends on the quality of the indicators, which in
turn depend on the underlying data. Managing data quality
throughout the process is thus crucial for a good assessment
and maximum comparability. When making comparisons a
transparent communication of all meta data underlining the
data sets is important, since it can explain how reliable the data
is – and thereby the results of the corresponding indicator(s).
Some data can’t be made open in its raw format due to
privacy protection, including citizen privacy, and
confidentiality issues. Cities still can have these data in their
internal systems following for each data set the access rules
and conditions specified in cities’ privacy protocols based on
national and European privacy regulations.
IV. PERFORMANCE MEASUREMENT SYSTEM
Together with the KPIs framework it was also developed a
prototype web based tool of the performance measurement
system integrating data input, calculation methodologies, and
720
result visualization in an intuitive and user-friendly interface.
Figure 2 shows an overview of this implemented Smart City
Performance Measurement System.
The most important parts of the prototype are
• The backend storing the CITYkeys indicator
framework, the city and project definitions, and the indicator
values.
• A RESTful API supporting e.g. querying, inserting,
and deleting the indicator values from the backend. This makes
it possible to utilise the CITYkeys backend with third party
software. The API enables cities to utilise CITYkeys
information in their own software tools or link the CITYkeys
platform with other city platforms.
• A user interface for indicator value input and
visualization.
• A service automatically reading supported open
datasets provided by the cities, calculating the related
indicators, and storing the calculated indicator values in the
backend. This is useful when cities update their datasets
regularly.
Fig. 2: The CITYkeys indicator framework
The user interface lets the user input new KPI values,
visualize previously entered values, and to download
previously entered KPI data as Excel spreadsheets.
The indicator selection user interface includes general
information about the studied city, the CITYkeys framework
for navigating between indicator subcategories and a list of all
CITYkeys city indicators including the latest previously
entered value, and a link to the value input page.
The KPI value input user interface includes the complete
description of the project/city indicator at hand and allows the
user to enter the assessment time and date of the data, the
assessment value, the indicator performance level on 1-5 scale
(only for project KPIs and Likert scale city indicators), and
additional information such as data sources, comments etc.
The KPI tool includes two different types of indicator
visualization, a “spider” diagram based visualization (Figure 3)
and a trend graph based visualization (Figure 4). The former is
for KPIs assessed on a uniform 1-5 performance scale to
illustrate the overall assessment results, and the latter for
individual KPIs analysis. The spider visualization supports
choosing the indicators to be visualized so that only the
relevant CITYkeys framework subcategories will be shown.
The tool also supports visualizing the indicators by
assessment time and comparisons between assessments made
at different times.
The trend graph based visualization is suitable for both
quantitative and performance level (Likert) indicators. It is
most relevant for automatically read indicators and others
having many timestamped values stored in the system (e.g. air
quality index and energy or water consumption related
indicators).
The concrete case studies in the five CITYkeys partner
cities, and 13 additional voluntary cities / other organisations,
have served in validating the indicator framework and testing
the tool prototype developed in CITYkeys. As a result,
recommendations have been made also on the use of the
framework and for refinements of some KPI descriptions. By
providing feedback and suggestions on the usability and
functionality of the early prototypes of the KPI tool cities have
participated in the co-design of the user interface at all stages
of the process.
Fig. 3: CITYkeys KPI tool: visualizing overall assessment
results
Fig. 4: CITYkeys KPI tool: trend graph visualization of
individual indicators.
V. NEW BUSINESS OPPORTUNITIES
The research work also focused on identifying new
business opportunities based on the use of the developed KPIs,
smart city performance measurement system, data collection
methods, and/or user interfaces. Especially service and
technology providers can make use of gaps in the current setup
721
to offer new solutions and meet the needs of stakeholders
involved.
Since market for performance measurement is already
established, to a certain extent, the identification was based
with the assessment of currently available services and
frameworks. Established neighborhood and city certification
services offer simple and widely understood solutions. Most of
the services are however only available for the project level or
for a certain policy field of the local authority. Services using
developed standards are often limited to either city or project
level. This leaves out one of the main needs of municipalities
to assess the contribution of a certain project or site
development to the city strategy. CITYkeys provides sets of
Key Performance Indicators for both levels allowing to show
the impact of projects.
Indices and assessment frameworks have easy-to-
understand results and allow for the comparison of different
environments. City representatives noted that these systems are
of low importance for their daily work since they do not
provide a level of detail that can be utilized by a municipal
administration. Many cities participate in public funded
research and demonstration projects. Research projects require
the establishment of a performance measurement system
(monitoring of performance and impact). The main
disadvantage is the short-term duration and difficulties in
replicability of the developed frameworks to other cities.
Solutions for open data management can be used to apply the
CITYkeys performance measurement framework or they can
serve as basis for the development of further applications for
citizens and businesses.
Several new opportunities that arise from the deployment of
the CITYkeys framework are related to consulting and
assessment services. Especially in the planning phase for new
developments and in the assessment of the contribution to city
strategies new services can be offered. Such services can be
offered either by a start-up or by established service providers
like consultancies. If an open data platform is in place the
CITYkeys indicator framework can be used to develop new
applications that make use of the data and of the developed
indicator set. One of the main outcomes of CITYkeys is the
performance measurement system. This is an overall solution
combining IT infrastructure and the indicator framework. The
implementation can require too much efforts to be
implemented by a small company. Alternative solution would
be a cooperative project, consortium or a use case in a business
incubator.
The main business opportunity arising from CITYkeys is
the implementation of an overall performance measurement
system. Development of an overall system requires different
professions to combine several approaches and technologies.
The result is an implemented open data platform providing
visualised results from calculated KPIs. The system can be
used for services described above (assessment, basis for the
development of new application). It can also offer a variety of
additional services. An overall CITYkeys performance
measurement system can be used to support business
development – publicly accessible results allow seeing what is
required in a city. Another possible application is the alignment
of local market. The performance measurement system can be
used as an application to establish common language between
stakeholders (using KPIs that can be understood by most of the
stakeholders). For the municipal administration such system
can support the procurement procedures. In this case
performance measurement is used as one of the information
sources for the setup of procurement procedures. The
procurement requirements can be adjusted on the actual
situation and the need to meet city targets.
VI. CONCLUSIONS
The main target groups of these results are cities and policy
makers who could use the KPI framework and the associated
tool in future. In addition, smart city project consortia in R&D
projects (particularly the H2020 SCC lighthouse projects) are
expected to use the CITYkeys framework in evaluating their
projects’ impact. For these projects, it would be especially
important to combine the typical input/output reporting (of e.g.
number apps of sensors implemented in a project) with
reporting on the impact achieved. Some lighthouse project
consortia have already implemented or engaged in CITYkeys
assessment in their projects but it is not yet the case for all of
them.
The main results from the extensive testing, druing
CITYkeys framework development can be summarised as
follows:
• Cities’ active involvement at all stages of the
development (of both KPI framework and performance
measurement tool) has been crucial for the quality and
usefulness of the results.
• The KPI tool developed has been successfully tested
and well received. In addition to the user interface testing, an
automatic data reading functionality has been validated. Cities
can also by themselves connect their own datasets to the tool
though the tool APIs.
• Most of the project and city KPIs have been tested in
several case studies with each having a different aim and focus.
The data availability and successful implementation of most of
the project KPIs (73/101 = 72%) and city KPIs (62/76=82%)
have been validated. The average KPI data availability rates in
a European city are expected to be over 70% (around 25% as
open data) for quantitative city KPIs and close to 100% for all
the qualitative ones.
• Several improvements in KPIs have been suggested
and, implemented.
• The framework structure and KPIs correspond well to
the key objectives of cities’ smart city projects, and along with
the well-received balance of qualitative and quantitative KPIs,
the assessment methodology has been validated as useful in
practice for smart city (project) assessment.
• A flexible approach in applying the KPI framework
based on city/case specific aims and priorities has been found
reasonable when using the KPIs/framework for different
722
purposes. That allows selecting only the most relevant KPIs for
assessment while a comprehensive and holistic assessment is
still encouraged.
• The main barrier in data collection, is not the data
availability or lack of expertise, but rather the localization and
accessibility of the needed data. The localization of the data
within or outside the often scattered city organization dealing
with the wide topic of a smart city has proved to be often a so
burdensome task that it sometimes leads to giving up the whole
KPI evaluation process due to the time needed.
• The development of a centralized data management,
storing and publishing practices, for open data, would help a lot
in the localization and exploitation of the currently vast amount
of available city data.
• As a later step, the standardization of (open) data set
formats would further improve the data exploitation
possibilities. In addition these steps would greatly improve the
efficiency of city processes including management,
coordination and reporting of smart city activities.
• CITYkeys RESTful APIs are a good and easy way to
open CITYkeys data for other developers. In future, linked data
would probably be a good way to integrate different types of
open datasets in different URLs.
ACKNOWLEDGMENT
The work has been co-funded by the European Commission
within the H2020 Programme. (Grant Agreement no: 646440).
Acknowledgment are due to all project partners: City of
Tampere, City of Zaragoza, City of Wien, City of Zagreb, City
of Rotterdam, TNO - Netherlands Organisation for Applied
Scientific Research, AIT Austrian Institute of Technology
GmbH; EUROCITIES ASBL and; VTT Technical Research
Centre of Finland.
REFERENCES
[1] European Commission, 2013, Report for the European Parliament:
Mapping Smart Cities in the EU. IP/A/ITRE/ST/2013-02
[2] OECD, 2012, OECD Environmental Outlook to 2050, OECD
Publishing.
[3] SBA Sustainable Building Alliance, 2009. Common Carbon Metric. For
Measuring Energy Use & Reporting Greenhouse Gas Emissions from
Building Operations. UNEP SBCI.
http://www.sballiance.org/dldocuments/common-carbon-
metric2009.pdf.
[4] EPA, United States Environmental Protection Agency, 2014, What is
sustainability?, http://www.epa.gov/sustainability/basicinfo.htm
[5] Giffinger, R., Fertner, C., Kramar, H., Meijers, E., Pichler-Milanoviü,
N., 2007, Ranking of European medium-sized cities, Final Report,
Vienna, 2007.
View publication stats
[6] Gonzales, J.A. and Rossi, A., 2011, New trends for smart cities, open
innovation mechanism in smart cities, European commission with the
ICT policy support programme.
[7] European Commission, 2014, Climate action.
http://ec.europa.eu/clima/policies/2030/index_en.htm
[8] Nam, T. & Pardo, T.A., 2011, Conceptualizing Smart City with
dimensions of technology, people and institutions. In 12th Annual
international conference on digital government research, 12-15 June,
College Park, MD.
[9] Hollands, RG. 2008. Will the real smart city please stand up? City 12, 3,
(December 2008), 303-320. DOI=http://10.1080/13604810802479126
[10] Allwinkle, S. and Cruickshank, P. 2011. Creating smarter cities: an
overview, Journal of urban technology, 18, 2,(April 2011), 1-16.
DOI=http://10.1080/10630732.2011.601103
[11] Malek, JA. 2009. Informative global community development index of
informative smart city, Proceedings of the 8th WSEAS (World Scientific
and Engineering Academy and Society) international conference on
education and educational technology. 17 – 19 October 2009, Genova,
Italy. ISSN: 1790-5109.
[12] European Commission, 2013, EIP SCC, European Innovation
Partnership on Smart Cities and Communities, Strategic Implementation
Plan, 14.10.2013, http://ec.europa.eu/eip/smartcities/
[13] Giffinger & Haindl 2009. Smart cities ranking: An effective instrument
for the positioning of cities?
https://upcommons.upc.edu/bitstream/handle/2099/11933/05_PROCEE
DINGS_M5_01_0014.pdf
[14] Canadian Environmental Assessment Agency. 2013. Cumulative Effects
Assessment Practitioners’ Guide. http://www.ceaa-
acee.gc.ca/default.asp?lang=En&n=43952694-
[15] GSMA 2013. Guide to Smart Cities. The Opportunity for Mobile
Operators. http://smartcitiesindex.gsma.com/indicators/
[16] Colldahl, C., Frey, S., Kelemen, J. E., 2013, Smart Cities: Strategic
Sustainable Development for an Urban World. School of Engineering.
Blekinge Institute of Technology. Karlskrona, Sweden.
[17] Lövehagen, L., Bondesson, A., 2013. Evaluating sustainability of using
ICT solutions in smart cities – methodology requirements. ICT4S 2013,
International Conference on Information and Communication
Technologies for Sustainability. pp. 175-182
[18] Ahvenniemi H., Huovila A., Pinto-Seppä I., Airaksinen M.. What are the
differences between sustainable and smart cities? Cities, Volume 60,
Part A (2017) 234–245.
[19] Airaksinen M., Ahvenniemi H., Virtanen M., 2012, Smart City Key
Performance Indicators, European Energy Research Alliance, EERA,
Join Program Energy in Cities status Report
[20] SCOPE, 2007. Sustainability Indicators: A Scientific Assessment.
Edited by T. Hák, B. Moldan and A.L. Dahl. Washington: Island Press.
2
[21] Kolk, A., 2004. “A Decade of Sustainability Reporting: Developments
and Significance.” International Journal of Environment and Sustainable
Development 3, no. 1 (2004): 51-64.
[22] Fortune, Joyce and Diana White, 2006. Framing of project critical
success factors by a systems model. International Journal of Project
Management 24 (2006) 53–65.
[23] Hiremath, R.B., Balachandra, P., Kumar, B., Bansode, S.S. and Murali,
J., 2013. Indicator-based urban sustainability—A review. Energy for
Sustainable Development. 17 (6), pp. 555–563.
723