Journal of Cleaner Production 233 (2019) 1269e1283

Contents lists available at ScienceDirect

Journal of Cleaner Production

journal homepage: www.elsevier.com/locate/jclepro

A critical review of selected smart city assessment tools and indicator

sets
Ayyoob Sharifi
Hiroshima University, 1-3-1 Kagamiyama, Higashi-Hiroshima, Hiroshima, 739-8530, Japan

a r t i c l e i n f o a b s t r a c t

Article history: In the era of digital revolution many cities around the world have invested significantly in the design and
Received 29 March 2019 implementation of smart city projects and initiatives to provide solutions to the challenges of climate
Received in revised form changes and urbanization. At the same time, various efforts have been made to evaluate performance
15 June 2019
and outcomes of those projects and initiatives. This study provides a critical analysis of 34 selected smart
Accepted 16 June 2019
city assessment tools to highlight their strengths and weaknesses and to examine their potential
Available online 17 June 2019
contribution to the evolution of the smart city movement. The selected tools are evaluated against an
^as de
Handling editor Cecilia Maria Villas Bo analysis framework that covers criteria related to comprehensiveness, stakeholder engagement, context
Almeida sensitivity, strategic alignment, uncertainty management, interlinkages and interoperability, temporal
dynamism, flexibility, feasibility, presentation and communication of the results, and action plans. Re-
Keywords: sults indicate that selected tools have achieved limited success in addressing these criteria. In particular,
Smart city only few tools have addressed criteria related to stakeholder engagement, uncertainty management,
Assessment tool interlinkages, and feasibility. The paper argues that assessment tools should capitalize on the advance-
Indicator
ments in smart solutions and big data analytics to develop better strategies for addressing these criteria.
Critical analysis
In addition to highlighting weaknesses that need to be addressed in the future, results of this study can
Sustainability
Climate change mitigation and adaptation be used by interested target groups such as smart city developers, planners, and policy makers to choose
tools that best fit their needs.
© 2019 Elsevier Ltd. All rights reserved.

1. Introduction
Cities have traditionally been the nuclei of change, innovation,
and economic development. As the global urbanization trends
continue, urban authorities face increasing challenges in delivering
the rapidly evolving citizen demands, while dealing with the
challenges and complexities of global sustainability (Clarke, 2013;
NIST, 2018; Stratigea et al., 2015). This ever-expanding scope and
magnitude of citizen demands and sustainability challenges ne-
cessitates shifting from traditional to more innovative and efficient
forms of urban planning and management.
Recognizing this need, cities around the world are increasingly
becoming dependent on Information and Communication Tech-
nologies (ICTs) to develop smart solutions for enhancing efficacy
and efficiency of service provision and management and for
advancing sustainability solutions in cities (Ahvenniemi et al., 2017;
Angelidou, 2015; Berst et al., 2014; Clarke, 2013; Huovila et al.,
E-mail addresses: sharifi@hiroshima-u.ac.jp, sharifigeomatic@gmail.com.
2019; Kirova and Mezzino, 2013; Kourtit and Nijkamp, 2018; Lee
et al., 2014; Lu et al., 2015; Naphade et al., 2011; Woods et al.,
2017). In line with this trend, the smart city concept has prolifer-
ated since the late 2000s, and many smart city projects have been
developed ever since (Ahvenniemi et al., 2017; Angelidou, 2015;
Caragliu et al., 2011; Marsal-Llacuna et al., 2015; Monfaredzadeh
and Berardi, 2015). This trend has intensified in the last few years
and is expected to continue in the future (Angelidou, 2015). Various
reasons can be mentioned for the growing interest in smart city
projects. Among others, smart city projects are deemed essential:
to gain and maintain competitive edge in a globally interconnected
economy (Giffinger et al., 2007a,b; Giffinger et al., 2010); to appeal
to the most talented, skilled, and creative citizens; to overcome
sustainability challenges and resource limitations that necessitate
efficiency improvements (Angelidou, 2015; BSI, 2014); to
contribute to climate stabilization by speeding up transition to low-
carbon society (Huovila et al., 2016); to improve transparency of
urban management (BSI, 2014); to improve Quality of Life (QOL)
and deal with multiple socio-economic challenges such as
inequality, insecurity, unemployment, and aging population

https://doi.org/10.1016/j.jclepro.2019.06.172
0959-6526/© 2019 Elsevier Ltd. All rights reserved.
1270 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283

(Manville et al., 2014); and overall, to make strides towards
achieving the Sustainable Development Goals (SDGs).
As the number of smart city projects continues to increase, there
is also a growing trend in the development and implementation of
tools and indicator sets for assessing their performance (Caird et al.,
2016). This is partly explained by the performance monitoring and
reporting requirements of funding bodies (Caird and Hallett, 2018),
and partly by multiple benefits that Smart City Assessment (SCA)
activities may offer to different actors and stakeholders. Some of
these benefits are summarized in Table 1.
While assessing smartness of cities has received more attention
in the recent years, few studies have analyzed SCA tools and their
strengths and weaknesses (see Section 2). To fill this gap, this study
provides a critical review of 34 selected SCA tools. The specific
objectives are: (a) to establish a set of criteria for evaluating con-
tent- and structural-validity of SCA tools and (b) to examine 34
selected tools using the evaluation criteria to understand their
strengths and weaknesses and to provide recommendations for
improving their performance.
The study is important because SCA is a nascent field with much
potential for future development. Having a better understanding of
SCA tools, their contents and structure, and their strengths and
weaknesses is critical for their future improvement and evolution.
Results of this study can inform interested actors of potential im-
provements needed to make SCA tools more effective for guiding
cities towards smartness. In addition, results can serve as a refer-
ence for different interested stakeholders to choose tools that best
serve their needs.
The paper is structured as follows. A brief literature review is
presented in the following section. Section 3 presents a brief
overview of the structure of the assessment tools and provides
details on research methods and materials. Results of the critical
analyses are presented in Section 4. Finally, the last section offers
reflections and concluding remarks.
2. Literature review
2.1. The smart city concept
There is no single, universally accepted definition for the smart
city concept and interpretations may vary depending on the
context (Li et al., 2019; Nam and Pardo, 2011). The concept com-
bines, builds on, and extends earlier paradigms such as digital city,
virtual city, ubiquitous city, intelligent city, creative city, knowledge
city, hybrid city, information city, and wired city (Albino et al., 2015;
Caird, 2017; Manville et al., 2014; Nam and Pardo, 2011; Zygiaris,
2012). With the evolution of the smart city concept over the past
decades, there has been a paradigm shift from an initially
technology-driven focus (with high emphasis on ICT) that is aimed
at maximizing efficiency of the hard urban infrastructure (i.e.,
transport, communications, waste, energy, water, etc.) to adopting
a more comprehensive approach, wherein the central role of people
and the soft infrastructure (i.e., institutions, citizen engagement,
data, social innovation, knowledge economy, justice, etc.) is
acknowledged (Ahvenniemi et al., 2017; Angelidou, 2014; Meijer
and Bolivar, 2016; Stratigea et al., 2015). By integrating multiple
dimensions, smart cities intend to function as agents of strategic
transformation and provide resources and pathways for increasing
operational efficiency, competitiveness, and QOL, and for
approaching sustainability (Ahvenniemi et al., 2017; Stratigea et al.,
2015). Therefore, while technology is the cornerstone and the
starting point for developing and advancing smart cities (Chourabi
et al., 2012; Debnath et al., 2014; Degbelo et al., 2016; Meijer and
Bolivar, 2016), appropriate consideration of other dimensions and
their interlinkages is essential for effective delivery and for im-
mediate and long-term success of smart cities (Manville et al., 2014;
Meijer and Bolivar, 2016). In fact, ICT is expected to not only
significantly strengthen interlinkages by providing interconnected
and interoperable platforms for multi-agent and multi-stakeholder
engagement, but also to improve the capacity to better understand
their underlying dynamics (ISO, 2014).
2.2. A brief overview of existing research on SCA
There is limited research analyzing SCA tools. This could be
explained by the fact that SCA itself is a relatively new field of
research and practice. In principle, two major approaches to the
study of SCA tools can be distinguished: one, focused on providing
an overview of the tools; and another one, involving more detailed
analyses of the tools to better understand their thematic focus and

Table 1
Potential multiple benefits of SCA for different stakeholders.

Stakeholder Potential benefits

Cities and city authorities - Monitoring performance to improve international image and competitive position of the city in the eyes of investors as well as creative
citizens and the public (Airaksinen, 2016; Giffinger et al., 2007; Giffinger et al., 2007a,b; Giffinger and Gudrun, 2010; Mohan et al., 2017)
- Justify the value of smart city investments and interventions (Caird et al., 2016)
- Identify strengths and weaknesses and guide smart city planning (Garau and Pavan, 2018; Giffinger et al., 2007a,b; Zygiaris, 2012)
- Track progress in achieving pre-defined goals and targets and identify position of the city in its efforts towards smartness (Garau and Pavan,
2018; SCR, 2014)
- Understand socio-economic and environmental implications of smart city projects (Sang et al., 2015)
- Understand technical requirements of smart city projects (Sang et al., 2015)
- Learn from the experiences of peers (when assessment involves benchmarking) (Debnath et al., 2014; Giffinger et al., 2007a,b; Giffinger et al.,
2010)
- Identify and showcase best-practice cases to learn lessons from (Giffinger et al., 2007a,b)
- Enhance governance transparency (Debnath et al., 2014; Giffinger et al., 2007a,b; Giffinger et al., 2010)
- Stimulate discussions among various stakeholders that may result in improved mobilization of resources (Barsi, 2018; Giffinger et al., 2007a,b)
Investors and funding - Evidence-based evaluation of the completed or ongoing projects (Caird et al., 2016)
agencies - Scientific mean for prioritizing funding allocation (Mohan et al., 2017)
- Enhanced ability to make decisions regarding the best sites for future investment (Giffinger et al., 2010; Mohan et al., 2017)
- Ability to identify and capitalize on new business opportunities (Airaksinen, 2016)
Researchers - Develop new strategies for improvement of smart city performance (Caird et al., 2016)
- Simplify the complexities of the smart city concept (Akande et al., 2019)
Citizens - Enhanced awareness about the benefits of smart city projects (Caird et al., 2016)
- Ability to make informed decisions when it comes to future investments (Mohan et al., 2017)
- Motivation to engage in smart city development activities and to communicate their desires and priorities to city authorities (Airaksinen,
2016; Debnath et al., 2014; Mohan et al., 2017)
 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
the typology of their indicators.
Related to the first approach, Albino et al. (2015) provides a brief
overview of tools and frameworks such as the smart ranking sys-
tems developed by the university of Vienna, the Intelligent Com-
munity Forum's Smart 21 communities, the Global Power City
Index, the Smarter Cities Ranking, the World's Smartest Cities, the
IBM Smart City, and the McKinsey Global Institute rankings. How-
ever, detailed analyses of these tools are not provided. Other tools
and frameworks such as the Smart City Maturity Model, the Smart
City Reference Model, the European Smart Cities Ranking model,
the Smart City Index Master Indicators framework, the Ericsson
Networked Society City Index, the Smarter City Assessment Tool,
and the Cities of Opportunity Index have been introduced by Caird
(2017).
There is, relatively, more research on the thematic focus and
typology of indicators. Stratigea et al. (2017) explored five inter-
national frameworks to compile a more exhaustive list of indicators
for smart-sustainable cities. They then classified the indicators into
six themes (i.e., economy, mobility, environment, people, living,
and governance) and found that most of them are related to the
‘living’ and ‘environment’ themes. In contrast, fewer indicators
were identified for the ‘governance’ and ‘people’ themes. They also
found that there is a lack of ICT-enabled indicators across the
selected assessment frameworks. In a similar effort to examine the
extent of inclusion of smartness indicators in assessment tools, Liao
et al. (2017) investigated eight, mainly Chinese, indicator sets and
found that they do not appropriately cover smart city dimensions
and, therefore, are not suitable for informing smart city planning.
Another Chinese team provided an overview of several assessment
frameworks but did not give details on their contents and structure
(Wu et al., 2016). Instead, they explored the approaches towards
indicator selection and found that most tools use expert-oriented
(based on the analytic hierarchy process (AHP)) approaches that
make the process subjective and non-participatory (Wu et al.,
2016).
Some studies have provided further details related to the ty-
pology and thematic distribution of indicators. For instance, a
meticulous study of seven indicator frameworks by Huovila et al.
(2019) showed that there is a lack of balance between sustain-
ability and smartness indicators, with the former being dominant
in most frameworks. Introducing five indicator types (i.e., input,
process, output, outcome, impact), their analysis showed that only
a small number of ‘process’ and ‘impact’ indicators have been used
for evaluating smartness and the dominant types are ‘output’ and
‘outcome’ (Huovila et al., 2019). A closely-linked study, examining
eight smart city indicator sets showed that social and economic
indicators are dominant, while there is a lack of attention to envi-
ronmental indicators (Ahvenniemi et al., 2017). The authors argued
that this is unlike sustainability indicator sets, where environ-
mental indicators are dominant (Ahvenniemi et al., 2017). Similar
findings have been reported in a study that shows SCA tools focus
on ‘people’ and ‘living’, while sustainability assessment tools
emphasize ‘environmental’ indicators (Monfaredzadeh and
Berardi, 2015).
Overall, two main gaps can be highlighted from this brief review
of the literature. First, the reviewed studies have mainly analyzed a
limited number of tools. The recent increase in the number of
assessment tools warrants including more tools in the analysis.
Second, existing research is mainly focused on thematic focus and
indicator type and many other important issues related to assess-
ment tools (see Section 3.3) have been overlooked. This study fills
these gaps by including a larger number of tools in the analysis, and
also by developing and utilizing an evaluation framework (that
includes factors other than just thematic focus and indicator type)
for a more comprehensive analysis of SCA tools.
1271
3. Methods and materials
3.1. Selection of the tools
An extensive search strategy was used to ensure inclusion of a
wide variety of tools available for assessing smartness at different
scales ranging from projects, to neighborhoods, communities, cit-
ies, and to city regions. The search string (see the Appendix No. 1)
included different terms such as ‘tool’, ‘toolkit, ‘index’, ‘framework’,
and ‘indicator set’ to acknowledge the fact that different developers
may use different terms when referring to assessment tools. Using
the ‘advanced search’ function of the Web of Science, the search
string was utilized to retrieve all types of English-language litera-
ture that match the search criteria. The initial search was per-
formed in July 2018 and returned 175 documents. After excluding
non-related categories, 58 documents remained in the database.
Those were screened for inclusion of assessment tools. In addition,
different combinations of the search terms were used to search for
grey literature in Google (first 20 pages). At the end of this process,
an initial list that included 38 tools was obtained. Final selection of
the assessment tools that are suitable for inclusion in the study was
based on the following criteria:
- Tools should have a clear focus on SCA;
- They should include a set of standards, metrics, or indicators
that can be used for SCA;
- They should cover more than one aspect/theme of smartness
(e.g., schemes that only cover mobility are excluded);
- Basic information related to the tools should be available in
English;
- Guidelines and manuals related to the tools should be freely
accessible (or the developer be willing to provide free access for
research purpose)
Thirty-four2 assessment schemes met these criteria. A complete
list of these tools is presented in Table 2. The table also includes
information on the release year and the primary developer(s). A
brief overview of these tools’ common approaches towards
assessing smartness is provided in the following section.
3.2. A brief overview of the common assessment steps
Like assessment tools developed for other purposes (Sharifi,
2016; Sharifi and Murayama, 2013, 2015), indicators are essential
constituent elements of SCA tools. Identifying a set of relevant in-
dicators is often the first step towards developing an assessment
tool. Indicators can be selected using one or a combination of the
following methods: literature review, expert survey, and stake-
holder consultation. Based on their similarity, selected indicators
are often classified into themes and sub-themes. An example of
commonly-used themes and sub-themes is shown in Fig. 1.
Evaluating performance against the indicators is done using
scorecards (e.g., in the SCC and WWC tools) that are often available
in the form of checklists, questionnaires, or other forms of score
sheets. Performance assessment can be conducted either by the
tool user (i.e., self-assessment, as in the CSC tool), by the tool
developer (e.g., in the EDCi and LRSC tools), or by external auditors
(i.e., third-party assessment, as in the SCIeI tool). Many tools
aggregate the score of individual indicators to obtain an aggregate
index that can be used to indicate the overall performance (e.g., the
SCIeI and CKPI tools). Since different types of variables may be used
2
Initial list included only 32 tools. However, two more tools were added during
the process.
1272 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283

Table 2
The list of assessment schemes analyzed in this study.

Tool Acronym Year Primary developer (s) Ref.

Lisbon ranking for smart sustainable cities LRSC 2019 Akande et al. (2019) Akande et al. (2019)
Smart Sustainable Cities China SSCC 2019 Li et al. (2019) Li et al. (2019)
Cities in Motion Index CIMI 2018 Center for Globalization and Strategy and IESE Business Berrone et al. (2018)
School's Department of Strategy
Global Power City Index GPCI 2018 The Mori Memorial Foundation's Institute for Urban Strategies MMF (2018)
Innovation Cities™ Index ICI 2018 China Academy of Telecommunication Research and China ICI (2018)
Communications Standards Association
EasyPark EP 2018 EasyPark group EasyPark (2018)
IOT-Enabled Smart city framework IES-City 2018 National Institute of Standards and Technology NIST (2018)
Smart Cities Council's tools and frameworks SCC 2018 Smart Cities Council, Australia and New Zealand (Berst et al., 2014; SCC,
2018)
What Works Cities WWC 2018 Bloomberg Philanthropies (BP, 2018)
Code for Smart Communities CSC 2018 Smart Cities Council Australian New Zealand and the Green CSC (2018)
Building Council of Australia
China Smart City Performance CSCP 2018 Shen et al. (2018) Shen et al. (2018)
Smart City Governments SCG 2018 Eden Strategy Institute and ONG&ONG Pte Ltd. ESI (2018)
Assessing Smart City Initiatives for the Mediterranean cnica of Madrid (UPM).
ASCIMER 2017 Universidad Polite (ASCIMER, 2017; Fernandez-
Region Anez et al., 2018)
Smart Cities Index- India SCIeI 2017 Indian School of Business Mohan et al. (2017)
Juniper Research smart city frameworks Juniper 2017 Juniper Research Juniper (2017)
UK Smart Cities Index UK-SCI 2017 Navigant Research Woods et al. (2017)
CITYkeys CITYkeys 2016 Netherlands Organization for Applied Scientific Research Bosch et al. (2017)
(TNO)
Networked Society City Index NSCI 2016 Ericsson in collaboration with Sweco Ericsson (2016)
Cities of Opportunity CoO 2016 PricewaterhouseCoopers (pwc) PWC (2016)
Community KPIs for the IoT and Smart Cities CKPI 2016 Future Everything Hemment et al. (2016)
Gulf States Smart Cities Index GSSCI 2016 Navigant Research Woods et al. (2016)
European Digital Cities Index EDCi 2016 Nesta Bannerjee et al. (2016)
Smart City Strategic Growth Map SCSGM 2016 ESPRESSO, European Commission Walter and Woodling (2017)
City IQ Evaluation System City-IQ 2015 Wu et al. (2016) Wu et al. (2016)
International Data Corporation (IDC) Smart City Analysis IDC 2015 IDC Achaerandio et al. (2012)
Telecommunication and Standardization Sector of ITU-T 2015 ITU-T Focus Group on Smart Sustainable Cities (ITU-, 2016)
International Telecommunication Union (ITU)
United Nations Economic Commission for Europe-ITU Smart UNECE 2015 UNECE Committee on Housing and Land Management; ECE (2015)
Sustainable Cities Indicators eITU Environment Agency Austria, and ITU
Smart Cities Ranking of European Medium-sized Cities EU-MSC 2014 TU Vienna, in cooperation with the University of Ljubljana and Giffinger et al., 2007a,b
the TU of Delft
Boyd-Cohen Smart City Index Boyd- 2014 Boyd-Cohen Cohen (2015)
Cohen
Mapping Smart Cities in the EU MSC-EU 2014 RAND Europe, European Union (EU) Manville et al. (2014)
Smart City Maturity Model and Self-Assessment Tool SCMM 2014 The Scottish Government and Scottish Cities Alliance (SCR, 2014; Urbantide,
2016)
Smart City Profiles SCP 2013 Austrian Climate and Energy Fund and Environment Agency Thielen et al. (2013)
Austria
United Cities and Local Governments (UCLG) smart cities UCLG 2012 City of Bilbao and Committee of Digital and Knowledge-based Azkuna (2012)
study Cities of UCLG
Smart Cities Benchmarking in China SCBC 2012 China Academy of Telecommunication Research and China SCBC (2012)
Communications Standards Association

for assessing performance against the smartness indicators,
obtaining aggregate index scores often requires normalizing the
individual indicators scores. Commonly-used normalization tech-
niques are the ‘min-max’ technique and the ‘z-score’ method (Wu
et al., 2016). In addition, when calculating the aggregate index
score, some tool developers (e.g., the NSCI and LRSC tools) assign
different weights to different indicators to acknowledge their
different levels of significance (Giffinger et al., 2010). At the end of
the assessment process various methods can be used to report and
disseminate the results. These methods have been explained in
detail in Section 4.10.
3.3. Framework for analysis
The reviewed literature recommends that SCA tools should
feature the following qualities: comprehensively address multiple
dimensions of smartness (Caird et al., 2016; Dirks et al., 2009), be
developed and implemented in a transparent and participatory
manner (Ahvenniemi et al., 2017; BSI, 2014; Caird et al., 2016;
Hemment et al., 2016; Rinaldi, 2017), be relevant to the local
context, be aligned with strategic needs and priorities (Dirks et al.,
2009; Fernandez-Anez et al., 2018), take necessary approaches to
deal with future uncertainties (Caird and Hallett, 2018; Debnath
et al., 2014; Manville et al., 2014; Stratigea et al., 2015), take ac-
count of interlinkages between different indicators (Dirks and
Keeling, 2009; Dirks et al., 2009; Giffinger et al., 2010; Woods
et al., 2017), be able to measure changes across temporal scales
(Caird, 2017; Kourtit and Nijkamp, 2018; Naphade et al., 2011), be
adaptable to different scales and locations (Airaksinen, 2016; BSI,
2015; Caird et al., 2016; Naphade et al., 2011), take account of
technical and financial feasibility issues (NIST, 2018), be able to
present and communicate the results in effective manners
(Giffinger and Gudrun, 2010; Kourtit and Nijkamp, 2018), and lead
to development of smart city action plans and roadmaps (Berst
et al., 2014; Fernandez-Anez et al., 2018). Table 3 lists a set of
several measurable evaluation criteria that are used, in this study,
to examine whether the selected tools feature these eleven quali-
ties. Further details about these qualities and their evaluation are
provided in the next section. Unless otherwise mentioned, content
analysis of documents related to the selected tools (i.e., manuals,
 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283 1273

Fig. 1. The smart city indicator themes and sub-themes.

Table 3
The framework for analysis.

Quality/issue Evaluation criteria

Comprehensiveness The extent of inclusion of indicators related to different smartness themes (dimensions) and sub-themes (sub-dimensions) in the selected
tools
Stakeholder engagement Whether participatory approaches have been taken for development and implementation of the selected tools
Context-sensitivity Whether the selected tools take account of citizen needs and context-specific needs and challenges
Strategic needs Whether the selected tools are aligned with strategic needs and priorities
Uncertainty management Whether iterative processes have been adapted and future scenarios have been developed to take account of future uncertainties
Interlinkages and Whether interlinkages and interoperabilities between different indicators have been considered in the assessment process
interoperability
Temporal changes Whether selected tools track temporal changes
Flexibility Whether issues related to flexibility, scalability, and replicability have been considered by the selected tools
Feasibility Whether issues related to technical and financial feasibility have been considered by the selected tools
Presentation and Whether selected tools have taken appropriate approaches for effective presentation and communication of the results
communication
Action-oriented approach Whether assessment findings have been used for developing action plans and smart city implementation roadmaps

guidelines, scorecards, policy papers, assessment reports, and peer- developed in Microsoft Excel to compile data related to the evalu-
reviewed articles) was the main method for evaluating their per- ation criteria. Matrix rows and columns corresponded to selected
formance using the framework presented in Table 3. Matrices were tools and the examined criteria, respectively. After matrices were
1274 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
completed, whenever needed, statistical analyses were performed
to obtain result reported in the following section. It is worth noting
that, in order to ensure accuracy of the compiled data, two rounds
of content analyses were performed for each assessment tool and
minor corrections were made during the second round.
4. Results and discussion
4.1. Comprehensiveness
As mentioned earlier, smart solutions are expected to provide
multiple socio-economic, environmental, and institutional benefits.
Therefore, assessment tools should not have a narrow focus on
technological indicators. Instead, they should, ideally, cover mul-
tiple themes (Caird et al., 2016). In fact, comprehensive indicator
coverage may facilitate a systems-based approach that allows un-
derstanding how smart city solutions may influence and transform
different urban sub-systems (Dirks et al., 2009).
In order to examine the extent of compliance with the
comprehensiveness criterion, it was necessary to first compile a list
of indicators. Accordingly, an extensive literature review was con-
ducted to create a pool of indicators. The initial list was then
compared with the indicator sets used by each of the selected
assessment tools to add additional indicators if needed. This was
done to ensure that the final list is as comprehensive as possible.
The (final) list presented in the Appendix No. 2 was prepared after
removing redundant indicators and combining similar ones. The
selected smartness indicators have been classified into seven
themes and 44 sub-themes. The seven themes (i.e., economy,
people, governance, environment, living, mobility, data) have
persistently been used across the selected tools. Therefore, adopt-
ing them for indicator classification helps to maintain consistency.
Fig. 1 shows the indicator classification approach used for the
purpose of this study (see the Appendix No. 2 for more details).
To examine the extent of inclusion of the indicators in the
selected tools, matrices were developed in Microsoft Excel (with
rows and columns corresponding to indicators and tools, respec-
tively). Guidelines and manuals related to each selected tool were
analyzed to determine whether the selected indicators have been
included. Here, results related to the extent of inclusion of in-
dicators and also the distribution of indicators across the themes (of
each selected tool) are presented.
4.1.1. The extent of inclusion of indicators in the selected tools
There is a wide variation among the tools in terms of the extent
of inclusion of indicators (see Table 1 of Appendix No. 1 for more
details). On average about 23% of the compiled indicators have been
included. A large variation can also be observed regarding the
extent of inclusion of indicators related to the seven themes. While,
on average, 30% of ‘mobility’ indicators have been included, the
corresponding value for ‘economy’ indicators is 19%. Fig. 2 shows
that while some tools such as IES-City, SCC, CITYkeys, and ITU-T
cover a relatively large percentage, less than 10% of the compiled
indicators have been included in the following tools: WWC, CSCP,
and SSCC.
A more detailed analysis shows that large variations also exist,
regarding the extent of inclusion, at the sub-theme and indicator
levels (see Appendix No. 2). ‘Innovation’ is the sub-theme that has
received the most attention and innovation-related indicators have,
on average, been included in 44% of the tools. Other highly-
included sub-themes are ‘ICT accessibility’, ‘equity and justice’,
‘ICT infrastructure’, ‘energy resources’, and ‘ICT management’.
Obviously, there is more focus on ICT-related sub-themes. On the
contrary, the following sub-themes have received the least atten-
tion: ‘cosmopolitanism’, ‘flexibility of the labor market’, ‘reacting
(data-informed)’, and ‘finance’.
Similarly, more attention to ICT was observed at the indicator
level, with the following indicators being included in about 70% of
the tools: ‘availability of digital infrastructure’, ‘presence of online
civic engagement and feedback system’, and ‘socio-economic
accessibility to digital technologies’. In contrast, many indicators
have been included in less than 20% of the tools. These include
indicators such as ‘quality of school education’, ‘tourism impact
management’, ‘flexibility of the workforce’, and ‘inhabitants’ atti-
tude towards international agreements'.
It is worth noting that using a comprehensive list of criteria may
sometimes be neither desirable nor feasible. In fact, comprehen-
siveness may result in creating a long list of indicators that may not
necessarily be measurable. This is mainly since data collection is
often a resource-intensive process and may require having access
to and control over different sources across different jurisdictions.
Comparability of data is also an issue as data may have been
collected using different protocols (Caird, 2017; Caird and Hallett,
2018). Therefore, when there are difficulties in including a
comprehensive list of indicators, it is recommended to utilize
techniques such as the Principal Component Analysis (PCA) to
choose a relatively balanced and representative list of indicators
(Akande et al., 2019).
4.1.2. Distribution pattern of indicators
Some interesting results were found regarding the distribution
of smartness indicators in each of the selected tools (note that this
is different from the above-presented analysis. Here, distribution
pattern of each tool's indicators and not the extent of inclusion of
the compiled indicators is investigated). Fig. 3 shows that most of
the selected tools include indicators related to all of the seven
themes. However, governance and mobility are the only themes
that have been included in all selected tools. A wide variation can be
observed among the selected tools in terms of their thematic
emphasis. Overall, there is a lack of balanced distribution of the
indicators and, on average, more emphasis has been placed on
‘mobility’, ‘economy’, and ‘environment’ themes (see Table 4). In
contract, the themes ‘people’ and ‘data’ have received less atten-
tion. This contradicts earlier findings in the literature about the lack
of attention to environmental indicators (Monfaredzadeh and
Berardi, 2015) and could be explained by the fact that only a
limited number of tools have been included in the earlier research.
The lack of attention to the ‘people’ and ‘data’ themes is despite
their significance for successful deployment of smart cities. Indeed,
city smartness is a multifaceted construct and can only be achieved
by appropriately acknowledging all aspects of smartness. Accord-
ingly, SCA tools should adopt a broad understanding of smartness
and recognize the importance of people as end users of smart so-
lutions (Ahvenniemi et al., 2017; Angelidou, 2014; Meijer and
Bolivar, 2016; Stratigea et al., 2015). In addition, data is argued to
be a significantly valuable resource for smart cities (Airaksinen,
2016; Berst et al., 2014; Wu et al., 2016; Zygiaris, 2012). There-
fore, SCA tools need to appropriately use indicators related to data
management to ensure suitable measures have been taken for
collecting and analyzing data.
More detailed analyses were conducted to also understand
distribution pattern of smartness indicators across the sub-themes
of each assessment tool (see Fig. 1 of Appendix No. 1). To better
communicate the results, distribution pattern of indicators at the
sub-theme level has also been presented for each of the main
themes. For example, percentage distribution of the frequency of
indicators falling under each sub-theme of the ‘mobility’ theme is
shown in Fig. 2 of Appendix No. 1. Other results of these analyses
are shown in Fig. 3 through 9 of Appendix No. 1. As can be observed,
large disparities exist between the tools that can be explained by
 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283 1275

Fig. 2. The extent of inclusion of the compiled indicators in the selected assessment tools (%). See Table 2 for a glossary of the acronyms.

Fig. 3. Percentage distribution of the frequency of indicators falling under each main theme. The Y-axis shows how the constituent indicators of each selected tool, on the X-axis are
distributed across the seven smart city themes. See Table 2 for a glossary of the acronyms.

contextual differences, methodological variations, and the fact that 4.2. Stakeholder engagement
assessment tools have been developed for different purposes.
Various benefits have been mentioned in the literature for
Table 4
stakeholder engagement in the development and implementation
Descriptive statistics on the distribution of indicators related to each theme across of assessment tools. Participatory approaches are particularly
the selected tools. needed to improve the procedural transparency and to ensure that
Mean (%) Median (%) Min (%) Max (%) Std (%)
assessment tools and indicators are meaningful to the stakeholders
(Ahvenniemi et al., 2017; BSI, 2014; Caird et al., 2016; Ericsson,
Economy 18.2 16.7 0 38 9
2016; Hemment et al., 2016; Rinaldi, 2017; SCA, 2014). In addi-
People 6.44 6.56 0 17 4
Governance 13.2 12 2 33 8 tion, such approaches may provide opportunities for collaboration
Environment 18 20.3 0 33 11 between different actors involved in the development and oper-
Living 15.8 17.1 0 33 9 ationalization of smart solutions; thereby, providing learning
Mobility 21 20.5 7 43 8 benefits, stimulating citizen creativity, and offering potentials for
Data 7.29 2.82 0 87 15
1276 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
synergies and innovative ideas (Caird et al., 2016; Lee et al., 2014;
Rinaldi, 2017).
Participatory approaches also enhance the sense of ownership
among community stakeholders that is critical for uniting them
toward realizing common goals (Sharifi, 2016; Simpson, 2017). In
addition, stakeholder engagement facilitates knowledge and
resource sharing, thereby improving the prospects of timely
implementation of the assessment process, as well as, the delivery
of the associated smart plans and programs (BSI, 2014; Simpson,
2017). These theoretical arguments have been supported by
empirical evidence (from cases such as Korea's New Songdo, the
United Arab Emirates' Masdar City, and Portugal's PlanIT Valley)
showing the utility of bottom-up participatory approaches for
eventual success of smart city development and assessment pro-
cesses (Hemment et al., 2016).
In the context of smart city planning, stakeholder participation
is specifically important because service provision in a smart city
does not follow a top-down mechanism, wherein central/local
government is the main provider. Instead, service provision is
increasingly achieved in a collaborative and bottom-up manner and
different stakeholders participate in co-creating and co-delivering
services (BSI, 2014). Therefore, stakeholder engagement in the
development and implementation of assessment tools can be
considered as a step toward mainstreaming participatory ap-
proaches in smart city planning.
Participation can take on various formats such as interviews and
questionnaire surveys, focus group discussions, community work-
shops, and design charrettes. Analysis of the selected tools shows
that most of them have not been developed and implemented
based on participatory approaches (see Fig. 4). This corroborates
previous claims about the lack of participatory approaches
(Ahvenniemi et al., 2017; Sharifi, 2016; Wu et al., 2016). Literature
review and expert surveys are the dominant development
methods. Some tools have been developed in consultation with
selected stakeholder groups. This has been considered as partial
Fig. 4. The extent of compliance with criteria related to ‘stakeholder engagement’, ‘context sensitivity’, ‘strategic needs’, and ‘uncertainty management’ in the selected tools (for
source data related to the figure see Tables 2 and 6 of Appendix No. 1).
engagement in the analysis. Similarly, most tools have not been
implemented following participatory approaches (Fig. 4). They
have mainly been implemented by either the developer or third-
party auditing agencies (note that ‘NA’ refers to those tools that
have not yet been implemented).
While integration of citizen-led and participatory approaches is
desirable and recommended for better reflection of stakeholder
needs, it should not be forgotten that such approaches may be
budget- and time-intensive and not compatible with the short-
term horizon of budget-constrained investors, developers, and
authorities (Sharifi, 2016). Therefore, adopting long-term planning
and investment strategies may be one way to deal with issues
related to the uptake of participatory approaches. It is also likely
that smart solutions will provide opportunities for reducing the
resource-intensity of participatory approaches. Here, it should be
mentioned that advocating for participatory approaches should not
be interpreted as a rejection of top-down approaches that are often
desirable for efficient resource allocation and management and for
coordinating efforts at larger scales (Achaerandio et al., 2012;
Sharifi, 2016). Therefore, a combination of top-down and bottom-
up approaches is likely to be more desirable.
4.3. Context sensitivity
Each city is unique in its context, vision, challenges, and prior-
ities. These idiosyncrasies should be considered in the assessment
process (Dirks et al., 2009; Fernandez-Anez et al., 2018). However,
given the resource limitations of developers and local authorities, it
would not be always possible to develop unique assessment tools
for different cities. Possible ways to deal with this issue are to apply
adjustment weights to indicators, to use local benchmarks for
conducting assessment, and to design the assessment tool in a
manner that cities could add bespoke indicators if needed.
Weighting factors are often assigned when cities have different
priorities depending on their context-specific conditions. For
 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
instance, a city in a flood-prone area is likely to assign higher
importance to indicators related to ICT-enabled early-warning
systems. Local (varying) benchmarks are particularly useful when a
specific indicator is used in cities with starkly different conditions.
For instance, benchmarks for water consumption in a water-
strained city may need to be different from cities with adequate
access to ample water supply. Finally, bespoke indicators are
needed when an issue is only of relevance in a specific context. For
instance, indicators related to earthquake would only be needed in
earthquake-prone areas.
Analysis of the selected tools shows that about 32% have
considered context-relevance issues (Fig. 4). This has been mainly
done through assigning weighting factors (e.g., in the MSC-EU tool).
However, other approaches have also been adopted (e.g., additional
bespoke indicators in the UNECE-ITU tool, and local benchmarks in
the CoO tool). Majority of the selected tools do not consider locally-
specific conditions, raising concerns about their context-relevance
and the validity of their assessment results.
As another measure of context-sensitivity, it was also investi-
gated whether citizen needs have been considered in development
of assessment tools. About 20% of the tools have paid attention to
this issue through approaches such as reporting different rankings
based on the opinions of different community groups (e.g., man-
agers, visitors, and residents) in the GPCI tool, or using surveys to
understand citizen preferences in the SCIeI tool.
4.4. Strategic needs
Alignment with city vision and strategic targets is critical for
success of any plans and programs (Manville et al., 2014). Smart city
projects often span over sectoral boundaries, making it essential to
establish alignment between organizational goals and project
performance measurement targets. Furthermore, bridging the gap
between performance measurement systems and broader goals
and strategies of different stakeholders improves the prospects of
operationalization of planning/measurement goals (Brorstro €m
et al., 2018; Caird and Hallett, 2018). Alignment of SCA tools with
strategic needs and targets is particularly essential because, unlike
‘short-term mindsets’, ‘long-term mindsets’ can facilitate sustained
implementation of smart city plans and programs (Kourtit and
Nijkamp, 2018; Simpson, 2017). Alignment of assessment in-
dicators with broader city vision and strategic needs is also desir-
able in the sense that assessment results can inform planners and
policy makers about the potential contributions of smart cities to
achieving strategic goals. This, in turn, may provide opportunities
for informing planners and policy makers about the benefits of
integrating smart city solutions into the processes for planning and
development of transformative city vision and strategic targets in
response to emerging city mechanisms (Caird et al., 2016).
Accordingly, assessment results may even provide useful input for
guiding strategic directions and needs (Kourtit and Nijkamp, 2018).
In the meantime, involvement in such processes may provide op-
portunities to leverage further support for sustained planning and
implementation of smart cities.
Selected tools were analyzed to understand whether they have
complied with the ‘alignment with strategic needs’ criterion. Re-
sults show that this issue has been addressed to some extent, as
about 32% of the tools have taken measures to ensure alignment
with strategic needs and targets (Fig. 4 above; and Table 2 of Ap-
pendix No. 1). This has mainly been achieved through efforts to
evaluate performance of smart cities against strategic targets. For
instance, in the CKPI tool alignment between the smart city projects
and higher-level goals is considered. The CSC tool emphasizes
alignment with urban design protocols in Australia and New Zea-
land. Furthermore, some tools even consider alignment with
1277
higher-level strategic goals such as the Sustainable Development
Goals and the New Urban Agenda (UNECE-ITU), and the European
Union 2020 targets (MSC-EU).
4.5. Uncertainty management
Long-term planning efforts have traditionally been hindered by
the limited predictability of future conditions. This may further
exacerbate due to the effects of phenomena such as climate change
that are expected to increase future uncertainties (IPCC, 2007).
Given the uncertainties inherent in future projections, reliance on
fixed targets and baselines is likely not sufficient for performance
assessment. Instead, it is desirable to adopt evolutionary and iter-
ative assessment processes that involve regular update of targets
and baselines (Sharifi, 2016).
Another approach for recognizing shifting baselines and
enhancing the ability to develop well-informed long-term plans
could be developing prediction models and/or future scenarios. In
fact, smart solutions’ application of big data analytics may provide
opportunities to develop reasonably accurate models and/or sce-
narios that can reduce the effects of future uncertainties by
improving prediction capacities (Debnath et al., 2014). Modelling
and scenario making approaches could also be effective for dealing
with urban dynamics and for developing resilient and forward-
looking plans that are aligned with higher-level strategic goals
(Kourtit and Nijkamp, 2018; Stratigea et al., 2015).
Analysis of the selected tools shows that about half of them have
adopted an iterative approach (Fig. 4), indicating that they may, to
some extent, be capable of dealing with uncertainties. On the
contrary, results show that very few modelling and scenario-
making efforts have been taken (e.g., efforts in the CoO tool to es-
timate the 2030 conditions). As explained above, modelling and
scenario-making can provide multiple benefits and deserve further
attention.
4.6. Interlinkages and interoperability
It is increasingly recognized that cities are ‘systems of systems’
(Dirks et al., 2009; Naphade et al., 2011; Woods et al., 2017). This
implies that multiple sub-systems of the urban system interact
with and influence each other through multiple feedback loops. For
instance, patterns of energy generation and consumption have
implications for management of water resources and vice versa
(Sharifi and Yamagata, 2016). Therefore, interventions aimed at
improving performance of energy systems, without considering the
implications for water resources, may result in trade-offs (Dirks and
Keeling, 2009; Dirks et al., 2009; Sharifi and Yamagata, 2016).
Accordingly, adopting holistic ‘systems-of-systems’ approaches
that consider the entirety of the urban system and shift away from
silo-based approaches to dealing with urban issues is needed (Dirks
et al., 2009; Naphade et al., 2011).
Accordingly, assessment tools should acknowledge these
inherent complexities of urban dynamics and take account of
interlinkages and interdependencies of different urban services.
This is particularly important in the context of smart cities that are
expected to utilize Internet of Things (IoT) and big data analytics to
provide a significantly better understanding of complex urban
dynamics (Woods et al., 2017). In addition to providing a basis for
better-informed decision making, considering interlinkages and
interoperabilities may also stimulate innovation and trans-
formation by creating platforms for co-creation and exchange of
information and ideas among and between users and providers.
Such exchanges may also lead to creation of new values and ideas,
improve efficiency and productivity of the system, enhance service
accessibility, and create opportunities for upgrading performance
1278 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
(Lee et al., 2014).
Despite all these benefits, one major drawback of many
assessment tools (designed for assessing smartness, sustainability,
resilience, etc.) is that they fail to take account of complex inter-
linkages and interdependencies (Giffinger et al., 2010; Sharifi,
2016). Also, only few methods have been suggested for this pur-
pose. For instance, the Analytic Network Process (ANP) model has
been proposed to divide assessment indicators into clusters and
explore the interactions and feedback processes within and be-
tween the clusters (Barsi, 2018).
Analysis of the selected tools confirms claims in the literature
about the limited consideration of interlinkages. The issue has only
been considered by four tools (Fig. 5). For instance, in the ASCIMER
tool it is explained how challenges related to each dimension are
related to the others. Or in the CoO tool, interlinkages are consid-
ered by examining how different indicators are correlated with
each other. Some tools such as SCMM have partially addressed this
issue by taking account of possible synergies between different
actions. However, no details have been provided on the inter-
linkages between different indicators.
4.7. Temporal changes
Cities are not static entities and their underlying dynamics
constantly transform in response to various socio-economic and
environmental mechanisms. Therefore, SCA should provide infor-
mation on such transformations and determine the state of prog-
ress toward achieving planning goals and objectives (Caird, 2017;
Kourtit and Nijkamp, 2018). This requires regular update of base-
line measurements that can be either used for tracking changes
through longitudinal analyses or through examining how distances
to pre-determined targets change over time (Caird et al., 2016).
Tracking changes over time provides means to understand perfor-
mance changes in reaction to smart planning interventions/initia-
tives that are expected to constantly change urban dynamics (Barsi,
2018; Caird, 2017). It will enable planners and policy makers to
understand the extent of transition to smartness, to find out how
strengths and weaknesses have changed over time, to evaluate the
efficacy and efficiency of their actions, and finally to make adjust-
ments to the actions if necessary.
Results show that, in half of the tools, assessment has only been
conducted for a single time epoch and, therefore, the results can be
described as a snapshot in time that are not appropriate for un-
derstanding evolutionary patterns and directions (Fig. 5). Those
considering temporal changes (about 40%) have relied on
comparing baseline performance at different time intervals.
Fig. 5. The extent of compliance with criteria related to ‘interoperability’, ‘temporal dynamism, ‘flexibility, and ‘feasibility’ (for source data see Tables 3 and 7 of Appendix No. 1).
4.8. Flexibility
Assessment tools should be flexible enough to be applicable to
different scales, locations, and situations (Airaksinen, 2016; BSI,
2015; Caird et al., 2016; Naphade et al., 2011). Flexibility is desir-
able under different conditions. For instance, updates and im-
provements may be required due to changes in targets or priorities
or in response to transformations in city's capacity for generating,
compiling, and analyzing data (Caird et al., 2016). In fact, each city is
unique and not every city may need to/or have the capacity to
conduct assessment using a fixed set of indicators and following a
specific assessment method (Naphade et al., 2011). Therefore,
assessment tools should be designed in a flexible manner that al-
lows them to be tailored to the needs, capacities, and priorities of
end users.
Accordingly, while scalability is needed to gain efficiency and to
achieve multiplier effects (BSI, 2015; Clarke, 2013), flexibility
should be acknowledged as an essential pre-requisite quality for
scalability and replicability of smart city initiatives in contexts
other than their origin. As Barsi (2018) argues, in addition to being
scalable and efficient, it is needed to “preserve opportunities for
spontaneity, and serendipity”.
Results show that about 41% of the tools have paid attention to
the flexibility issue (Fig. 5). The flexibility criterion has mainly been
addressed through allowing evaluators to use proxy data when the
target data for specific indicators is missing. For instance, in the
SCIeI tool, average of scores for other indicators within the same
theme would be used as a proxy. Taking a different approach, the
‘Juniper’ and ‘CoO’ tools use regional or national data, as proxy data,
when city-specific data is missing.
Another commonly-used way for addressing the flexibility cri-
terion is to develop general assessment frameworks that allow
cities to identify locally-relevant domains and indicators in
collaboration with stakeholders. This has been practiced by tools
such as CKPI and SCR. Finally, some tools such as SCC and UNECE-
ITU allow the evaluators to choose indicators, out of the indicator
pool, that apply to their respective city. To maintain a certain degree
of consistency and to ensure meeting some minimum standards,
the UNECE-ITU includes two types of indicators: core and addi-
tional. Evaluators decide at their own discretion whether to include
additional indicators.
4.9. Feasibility
Budget limitations and technical constraints are two likely
barriers to the effective implementation of smart cities (NIST, 2018;
 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
Simpson, 2017). As costs can be prohibitive to the deployment of
smart cities, developers need to take appropriate strategies early in
the process to secure sustainable sources of funding. These could be
through delivering small demonstration projects to pave the way
for securing long-term investment (Simpson, 2017). Financial
feasibility could also be demonstrated through detailed cost-
benefit analyses that justify the short-, medium-, and long-term
costs and benefits of smart plans.
Regarding technical feasibility, appropriate measures should be
taken to ensure availability of enough managerial and technical
skills for proper implementation of smart city projects. This is
essential to avoid proposing ambitious and complex plans that may
not be deliverable due to the lack of technical and managerial skills.
Therefore, mapping technical and institutional capacities of
different actors involved in the planning and implementation
processes is needed.
Accordingly, it is suggested that assessment tools should involve
mechanisms for evaluating financial and technical feasibility of
proposed smart city plans and projects. Analyses of the selected
tools indicate that this issue has not yet been well-recognized, and
majority of the tools have not considered mechanisms for projec-
ting costs and benefits and for ensuring presence of technical
expertise (Fig. 5). There are only few exceptions. For instance, the
MSC-EU tool provides guidance on conducting cost-benefit analysis
to demonstrate monetary benefits and financial feasibility. Or, the
SCR tool emphasizes mapping skills needed for delivering smart
city projects.
4.10. Presentation and communication of the results
Upon completion of the assessment process, it is needed to
report the results in a timely and effective manner. Various factors
should be considered during this stage. First, it is important to
acknowledge that different end users may have different needs and,
as much as possible, results should be reported in various formats
to meet potentially diverse expectations.
For instance, some end users such as the public or media rep-
resentatives may only be interested to know the aggregate city
performance, and/or the overall ranking compared to other cities.
However, such generic results would not be useful for other groups
such as planners, policy makers, and researchers who need to know
more details related to the underlying research methodologies, the
performance of different themes and sub-themes (i.e., detailed
performance breakdown for each theme, sub-theme, and in-
dicators), strengths and weaknesses, and the state of progress.
Without such details it would not be possible to revisit the activ-
ities, address shortcomings, identify priorities, develop pathways
for achieving better performance in the future, and provide
appropriate policy advice (Giffinger et al., 2007a,b; Giffinger and
Gudrun, 2010; Kourtit and Nijkamp, 2018).
Ideally, results should be communicated with the end users
using various means such as media release and press conferences,
online performance dashboards, community workshops, and social
media. Such a diverse communication and dissemination approach
can provoke discussions among various stakeholders that may lead
to enhanced collaboration towards improved development and
implementation of smart city projects (Giffinger and Gudrun, 2010;
Giffinger et al., 2010). Engaging different stakeholders in the
reporting process may also facilitate learning, experience-sharing,
and capacity-building benefits (Caird et al., 2016; Giffinger et al.,
2007a,b). More specifically, it may enhance public awareness of
the multiple benefits of city smartness, thereby improve the pros-
pects of implementation. Communication have been reasonably
well addressed by some of the selected tools. For instance, the CKPI
tool uses methods such communication during iterative
1279
workshops, social media release, and communication using com-
munity champions as liaison agents.
Regardless of the communication type, an effective way for
reporting assessment results is using illustration techniques such as
spider diagrams, interactive graphs, bar diagrams, and scatter plots.
More than 50% of the selected tools have utilized such illustration
techniques.
Results show that providing overall ranking is the most common
method of reporting results (Fig. 6). In some cases, ranking is also
accompanied by labelling that makes it easier to differentiate be-
tween cities (e.g., the WWC tool uses the following labels: Plat-
inum, Gold, Silver). Ranking and labelling are particularly popular
for the purpose of performance benchmarking that, if conducted
appropriately, may encourage sense of competition among the
cities. However, uninformed and unreasonable comparison be-
tween cities, wherein winners capitalize on the results for mar-
keting purposes, while losers ignore the results, should be avoided
(Giffinger et al., 2007a,b; Giffinger and Gudrun, 2010). While overall
ranking has been widely used as a reporting method, less than 40%
of the selected tools provide sub-theme ranking and detailed per-
formance breakdown. Good examples are the City-IQ tool that
utilizes spider diagrams to provide detailed performance on each
indicator (on a 1e100 scale) for different cities (Wu et al., 2016), or
the CIMI tool's performance breakdown for different themes (see
Fig. 10 of Appendix No. 1).
Selected tools perform reasonably good in terms of showing the
strengths and weaknesses of smart cities (see Fig. 6). However,
there is a tendency towards highlighting strengths, and weaknesses
are not always well reported. Emphasizing positive results may be
due to the complexities of urban politics (Giffinger et al., 2010). A
good example of how to highlight weaknesses has been presented
in the EU-MSC tool (see Fig. 7).
Finally, as discussed in Section 4.7, assessment results should
show temporal changes and evolution patterns. It is desirable to use
illustration techniques to better communicate such changes and
patterns. Some useful examples that can be followed by other tools
have been presented in the Appendix No. 1 (Figs. 11 and 12).
4.11. Action plans
As mentioned earlier, medium- and long-term planning is
needed for developing smart plans. This is mainly because transi-
tion to a smart city may involves major transformations that do not
occur abruptly (e.g., due to society's natural tendency to resist
change). Developing a roadmap with clearly-defined milestones
and action plans is critical to overcome such barriers and can
contribute to a smooth transition to a desired future state (Berst
et al., 2014). Roadmapping may also provide co-benefits in terms
of mobilizing otherwise fragmented efforts and improving opera-
tional efficiency (Berst et al., 2014).
Connecting assessment results to action plans is essential to
effectively influence the planning and policy making processes. To
facilitate such a connection, in addition to highlighting gaps,
assessment process should involve prioritizing areas for interven-
tion and corrective action. Such information can be used by plan-
ners and policy makers for developing better-informed roadmaps
and action plans (Berst et al., 2014; Fernandez-Anez et al., 2018;
Sharifi, 2016).
Results show that more efforts are needed to connect assess-
ment results to action plans (Fig. 6). Only about 25% of the selected
tools provide recommendations on how to link assessment findings
to roadmapping and action planning. Of those, the SCC tool can be
highlighted for its detailed and step by step explanation on how to
leverage assessment results to create and implement roadmaps
(Berst et al., 2014; SCC, 2018).
1280 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283

Fig. 6. The extent of compliance with criteria related to ‘presentation and communication, and ‘action plans’ (for source data related to the figure see Tables 4e6 of Appendix No. 1).

Fig. 7. An example of reporting weaknesses of Luxembourg when compared to its peers in the European Smart Cities project (Giffinger et al., 2014). The tool developers have used
normalization techniques to calculate the theme-level performance of each city compared to its peers. For example, as can be seen, when compared to other medium-size European
cities, Luxembourg does not perform good in terms of smart governance.

5. Summary and conclusions
Over the past few years there has been a considerable increase
in the number of tools developed for assessing performance of
smart city projects and initiatives. In this study 34 tools were
analyzed to provide a better understanding of their strengths and
weaknesses in terms of contents, structure, and procedures taken
for SCA. An analysis framework was introduced that can be used by
interested target groups to evaluate performance of a SCA tool(s).
Results of such evaluations can be used for different purposes. For
instance, developers or city authorities may apply the analysis
framework for choosing tools that best fit their needs. Further,
researcher and tool developers may use the framework to examine
content- and structure-validity of SCA tools and to make adjust-
ments if needed.
Evaluating the selected SCA tools using this framework showed
that, overall, limited success has been achieved and several im-
provements are needed as the tools continue to evolve over the
coming years. A summary of the main strengths and weaknesses
related to the 11 analysis criteria is presented in Table 5. Also,
important planning/policy considerations and implications related
to these criteria are summarized in Table 6.
A major shortcoming that is worth highlighting again is the
limited use of modelling and scenario-making techniques for
dealing with future uncertainties. Such techniques are specially
needed to anticipate future changes and to avoid being over-
whelmed by constantly changing dynamics of the urban system. It
is hoped that application of big data analytics will provide more
opportunities for integration of modelling and scenario making
into the assessment process (Debnath et al., 2014). Use of big data
analytics and IoT is also expected to provide a significantly better
understanding of complex urban dynamics and to offer opportu-
nities for taking interlinkages between indicators into account
(Woods et al., 2017). In fact, while different urban sub-systems are
interlinked and should be treated as such, interlinkages have not
been appropriately addressed by the selected SCA tools. Using
smart solutions is, therefore, recommended to overcome siloed
barriers and cut across different subsystems of the urban system
 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283 1281

Table 5
A summary of the major strengths and weaknesses related to the analysis criteria.

Criterion/issue Strength(s) Weakness (es)

Comprehensiveness - The following tools provide a relatively good coverage of the smart city - Limited coverage of smart city indicators was observed in the
indicators: IES-City, SCC, CITYkeys, and ITU-T following tools: WWC, CSCP, and SSCC
- There is a good coverage of indicators related to innovation and ICT - Some indicators related to the ‘economy’ and ‘people’ themes
are not well-addressed
- There is a lack of balanced distribution of indicators
Stakeholder - The CKPI tool adopts a bottom-up, co-design approach and can provide insights - Most tools have not taken measures to engage stakeholders in
engagement on how to involve community members and different stakeholders their development and implementation processes
Context-sensitivity - Efforts have been made to address the issue through assigning weighting factors - Majority of the selected tools do not consider locally-specific
to the indicators (e.g., in the MSC-EU tool) conditions
Strategic needs - The following tools have taken measures to ensure alignment with strategic - The issue has not been well-addressed by the other tools
needs: CKPI, CSC, UNECE-ITU, and MSC-EU analyzed in this study
Uncertainty - Half of the tools have made efforts to deal with this issue by adopting an - Despite their utility, modelling and scenario-making ap-
management iterative assessment approach proaches have not been appropriately used
Interlinkages and - Few tools such as ASCIMER and CoO have taken measures to address this issue - There is a lack of efforts to utilize the potentials of IoT and big
interoperability data analytics for better understanding of interlinkages
Temporal changes - About 40% of the tools have compared baseline performance at different time - Reliance on single time epochs for assessing smartness is still
epochs the dominant approach
Flexibility - The issue has been dealt with by the following tools: SCIeI, CoO, CKPI, SCR, SCC, - The other tools lack strategies to address the flexibility issue
and UNECE-ITU
Feasibility - The MSC-EU tool uses cost-benefit analysis to examine financial feasibility - Overall, the issue of feasibility has not been well addressed
- The SCR tool includes strategies for evaluating technical feasibility across the tools
Presentation and - Communication techniques have been relatively well-used across the tools - Only less than 40% of the tools provide sub-category theme
communication - Strengths and weaknesses have been reported by more than 80% of the tools ranking and detailed performance breakdown
Action plans - The SCC tool provides a detailed procedure for linking assessment results to - Only about 25% of the tools provide recommendations on
roadmapping linking results to action plans

Table 6
Important planning/policy considerations and implications.

Criterion/Issue Planning/policy considerations and implications

Comprehensiveness - Smart city is a multi-dimensional concept and indicators related to different dimensions should be included in assessment tools
- Currently, more emphasis has been placed on ‘mobility’, ‘economy’, and ‘environment’ themes. A more balanced distribution of the
indicators is needed
- Only emphasizing innovation and ICT-related sub-themes is not enough and appropriate measures should also be taken to include indicators
related to relatively under-covered sub-themes such as ‘cosmopolitanism’, and ‘flexibility of the labor market’
- Including a comprehensive list of indicators may not always be feasible; techniques such as the Principal Component Analysis (PCA) can be
used to choose a relatively balanced and representative list of indicators
Stakeholder engagement - Limited success has been achieved in terms of adopting participatory approaches for development and implementation of assessment tools.
As explained in Section 4.2, such approaches can provide multiple benefits and deserve further attention
Context-sensitivity - Most of the selected tools have not been tailor-made to local conditions. Applying adjustment weights to indicators, using local benchmarks,
and allowing inclusion of bespoke indicators are among the measures that could be utilized to develop context-sensitive tools
Strategic needs - Only about one third of the selected tools have paid attention to alignment with strategic needs. More attention to this issue is needed since
it may not only sustain implementation of smart city projects, but also provide input for guiding strategic directions
Uncertainty management - In addition to iterative assessment processes, modelling and scenario making approaches should be adopted to better address issues related
to future uncertainties
- Better application of big data analytics is recommended as it may enhance the capacities to develop more accurate models and/or scenarios
that improve prediction abilities
Interlinkages and - Lack of ability to deal with the interlinkages of different sub-systems of the urban system is a major drawback of assessment tools. SCA tools
interoperability should tap into the opportunities offered by Internet of Things (IoT) and big data analytics to address this shortcoming
Temporal changes - In many tools, assessment has only been conducted for a single time epoch. Iterative assessment processes are needed to understand how
baseline conditions change in response to interventions
Flexibility - Flexibility is needed to ensure applicability of assessment tools to different contexts and under different conditions. Some recommended
solutions for improving flexibility include allowing cities to use locally-relevant indicators and using proxy data to substitute for missing
data
Feasibility - High costs and lack of managerial and technical skills could be prohibitive to the deployment of smart cities. Therefore, assessment tools
should involve mechanisms for evaluating financial and technical feasibility of proposed smart city projects
Presentation and - Assessment results should be reported in various formats to serve the needs and expectation of a wide range of stakeholders
communication - While ranking can be useful for performance benchmarking, appropriate measures should be taken to ensure that it instigates constructive
competition and is not only used, by the winners, for marketing purposes
- Only reporting overall ranking/performance is not enough and results related to the themes and sub-themes should also be reported to
highlight the strengths and weaknesses
Action plans - In addition to highlighting gaps and weaknesses, assessment should involve prioritizing areas for intervention and corrective action and
developing action for implementing them

(Albino et al., 2015; Caird, 2017; Lee et al., 2014).
Overall, this research was successful in delineating some
strengths and weaknesses of SCA tools. More research is needed to
gain further details about the suitability of individual SCA tools for
guiding cities towards smartness. Such research should be
informed by empirical findings from case studies to better
understand the complexities of smart city assessment. At the end, it
is worth noting that in many parts of the world, cities are still at the
early stages of developing and operationalizing SCA (Caird and
Hallett, 2018), and more importantly assessment practices are not
yet integrated into official urban planning and management
mechanisms and strategies (Caird and Hallett, 2018). This is partly
1282 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
because of the lack of statutory mechanisms to mainstream SCA
into the planning and decision-making processes (Caird and
Hallett, 2018). However, increasingly cities are required to annu-
ally report their performance on different sets of key performance
indicators and it is obvious that statutory requirements to report
performance on smartness may be seen as an additional burden
(Caird and Hallett, 2018). It is therefore, worthwhile to develop
methods for integrating smartness criteria into existing assessment
tools. Such tools would be capable of providing performance re-
ports on different aspects (e.g., smartness, sustainability, resilience,
etc.) in an efficient and integrated manner.
Declarations of interest
None.
Acknowledgements
I would like to sincerely thank three anonymous reviewers of
the journal for the constructive comments on the original draft of
the manuscript.
Appendix A. Supplementary data
Supplementary data to this article can be found online at
https://doi.org/10.1016/j.jclepro.2019.06.172.
References
Achaerandio, R., Bigliani, R., Curto, J., Gallotti, G., 2012. Smart Cities Analysis in Spain
2012eThe Smart Journey. http://www.portalidc.com/resources/white_papers/
IDC_Smart_City_Analysis_Spain_EN.pdf. visited 25 July 2013.
Ahvenniemi, H., Huovila, A., Pinto-Seppa, I., Airaksinen, M., 2017. What are the
differences between sustainable and smart cities? Cities 60, 234e245.
Airaksinen, M., 2016. Smart Cities, Can the Performance Be Measured? VTT Impulse.
VTT Finland, pp. 26e33.
Akande, A., Cabral, P., Gomes, P., Casteleyn, S., 2019. The Lisbon ranking for smart
sustainable cities in Europe. Sustain. Cities Soc. 44, 475e487.
Albino, V., Berardi, U., Dangelico, R.M., 2015. Smart cities: definitions, dimensions,
performance, and initiatives. J. Urban Technol. 22 (1), 3e21.
Angelidou, M., 2014. Smart city policies: a spatial approach. Cities 41, S3eS11.
Angelidou, M., 2015. Smart cities: a conjuncture of four forces. Cities 47, 95e106.
ASCIMER, 2017. Assessing Smart City Initiatives for the Mediterranean Region. EIB
University Research Sponsorship Programme.
BP, 2018. What Works Cities Certification Assessment Guide. https://
whatworkscities.bloomberg.org/certification/. Accessed 11/12 2018.
SCC, 2018. Smart Cities Guide for Built Environment Consultants. Smart Cities
Council Australia New Zealand and Consult Australia.
Azkuna, I., 2012. Smart Cities Study: International Study on the Situation of ICT,
Innovation and Knowledge in Cities. The Committee of Digital and Knowledge-
based Cities of UCLG, Bilbao.
Bannerjee, S., Bone, J., Finger, Y., Haley, C., 2016. European Digital City Index
Methodology Report. Nesta, London. https://digitalcityindex.eu/download.
Barsi, B., 2018. Beyond indicators, new methods in Smart city assessment. SCRD
Smart Cities Reg. Dev. J. 2 (1), 87e99.
Berrone, P., Ricart, J.E., Carrasco, C., T-Figueras, A.I.D., Giuliodori, D.A.,
Giuliodori, M.A., 2018. IESE cities in motion index. IESE.
Berst, J., Enbysk, L., Williams, C., 2014. Smart Cities Readiness Guide: the Planning
Manual for Building Tomorrow's Cities Today. Smart Cities Council, Seattle.
Bosch, P., Jongeneel, S., Rovers, V., Neumann, H., Airaksinen, M., Huovila, A., 2017.
CITYkeys Indicators for Smart City Projects and Smart Cities. CITYkeys Report.
Brorstro €m, S., Argento, D., Grossi, G., Thomasson, A., Almqvist, R., 2018. Translating
sustainable and smart city strategies into performance measurement systems.
Publ. Money Manag. 38 (3), 193e202.
BSI, 2014. Smart City Framework e Guide to Establishing Strategies for Smart Cities
and Communities, PAS 181:2014. The British Standards Institution.
BSI, 2015. Smart Cities Overview e Guide PD 8100:2015. BSI.
Caird, S., 2017. City approaches to smart city evaluation and reporting: case studies
in the United Kingdom. Urban Res. Pract. 11 (2), 159e179.
Caird, S.P., Hallett, S.H., 2018. Towards evaluation design for smart city develop-
ment. J. Urban Des. 1e22.
Caird, S., Hudson, L., Kortuem, G., 2016. A Tale of Evaluation and Reporting in UK
Smart Cities.
Caragliu, A., Del Bo, C., Nijkamp, P., 2011. Smart cities in Europe. J. Urban Technol. 18
(2), 65e82.
Chourabi, H., Nam, T., Walker, S., Gil-Garcia, J.R., Mellouli, S., Nahon, K., Pardo, T.A.,
Scholl, H.J., 2012. Understanding Smart Cities: an Integrative Framework, Sys-
tem Science (HICSS), 2012 45th Hawaii International Conference on. IEEE,
pp. 2289e2297.
Clarke, R., 2013. Business Strategy: IDC Government Insights' Smart City Maturity
ModeldAssessment and Action on the Path to Maturity. International Data
Corporation (IDC) Government Insights, Business Strategy# GI240620, Alex-
andria, VA: USA.
Cohen, B., 2015. The Smartest Cities in the World 2015: Methodology. https://www.
fastcompany.com/3038818/the-smartest-cities-in-the-world-2015-
methodology.
CSC, 2018. Code for Smart Communities. Smart Cities Council Australia and New
Zealand.
Debnath, A.K., Chin, H.C., Haque, M.M., Yuen, B., 2014. A methodological framework
for benchmarking smart transport cities. Cities 37, 47e56.
Degbelo, A., Bhattacharya, D., Granell, C., Trilles, S., 2016. Toolkits for Smarter Cities:
A Brief Assessment. Ubiquitous Computing and Ambient Intelligence, vol.
10070, pp. 431e436. Ucami 2016, Pt Ii.
Dirks, S., Keeling, M., 2009. A Vision of Smarter Cities.
Dirks, S., Keeling, M., Dencik, J., 2009. How Smart Is Your City?: Helping Cities
Measure Progress. IBM Institute for Business Value, IBM Global Business Ser-
vices, New York.
EasyPark, 2018. 2018 Smart Cities Index. https://easyparkgroup.com/smart-cities-
index/. Accessed 19/12/2018 2018.
ECE, 2015. The UNECEeITU Smart Sustainable Cities Indicators. Economic Com-
mission for Europe.
Ericsson, 2016. Networked Society City Index, 2016 edition (Ericsson).
ESI, 2018. Top 50 Smart City Governments. Eden Strategy Institute and ONG&ONG
Pte Ltd.
Fernandez-Anez, V., Velazquez, G., Perez-Prada, F., Monzo n, A., 2018. Smart city
projects assessment matrix: connecting challenges and actions in the medi-
terranean region. J. Urban Technol. 1e25.
Garau, C., Pavan, V.M., 2018. Evaluating urban quality: indicators and assessment
tools for smart sustainable cities. Sustain. Basel 10 (3), 575.
Giffinger, R., Gudrun, H., 2010. Smart cities ranking: an effective instrument for the
positioning of the cities? ACE - Archit. City Environ. 4 (12), 7e26.
Giffinger, R., Fertner, C., Kramar, H., Kalasek, R., Pichler-Milanovi c, N., Meijers, E.,
2007a. Smart Cities: Ranking of European Medium-Sized Cities. Centre of
regional science (srf), vienna university of technology, vienna, austria.
Giffinger, R., Fertner, C., Kramar, H., Meijers, E., 2007b. City-ranking of European
Medium-Sized Cities. Cent. Reg. Sci., Vienna UT, pp. 1e12.
Giffinger, R., Haindlmaier, G., Kramar, H., 2010. The role of rankings in growing city
competition. Urban Res. Pract. 3 (3), 299e312.
Giffinger, R., Kramar, H., HAINDLMAIER, G., STROHMAYER, F., 2014. European Smart
Cities 3.0. Benchmarking, 2014. http://www.smart-cities.eu.
Hemment, D., Woods, M., Appadoo, V., Bui, L., 2016. Community Key Performance
Indicators (Community KPIs) for the IoT and Smart Cities: A Collaborative
Framework for Project Assessment. FutureEverything.
Huovila, A., Airaksinen, M., Pinto-Seppa €, I., Piira, I., Penttinen, T., 2016. Smart City
Performance Measurement System, 41st IAHS World Congress on Housing:
Sustainability and Innovation for the Future. IAHS, 2016.
Huovila, A., Bosch, P., Airaksinen, M., 2019. Comparative analysis of standardized
indicators for Smart sustainable cities: what indicators and standards to use
and when? Cities 89, 141e153.
ICI, 2018. Innovation Cities™ Index 2018: Global. https://www.innovation-cities.
com/innovation-cities-index-2018-global/13935/. Accessed 19/12/2018.
IPCC, 2007. Climate Change 2007: Synthesis Report. Contribution of Working
Groups I, II and III to the Fourth Assessment Report of the Intergovernmental
Panel on Climate Change. IPCC, Geneva, Switzerland.
ISO, 2014. Smart Cities Preliminary Report : ISO/IEC JTC 1 Information Technology.
International Organization for Standardization (ISO).
ITU-T, 2016. Overview of key performance indicators in smart sustainable cities.
TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU.
Juniper, 2017. Smart cities: what's in it for citizens? Juniper Res.
Kirova, T.K., Mezzino, D., 2013. Landscape Design and Sustainable Development:
from Smart City to the Smart Land. Heritage, Architecture. Landesign: Focus on
Conservation, Regeneration, Innovation, pp. 1377e1383, 39.
Kourtit, K., Nijkamp, P., 2018. Big data dashboards as smart decision support tools
for i -cities e an experiment on stockholm. Land Use Policy 71, 24e35.
Lee, J.H., Hancock, M.G., Hu, M.C., 2014. Towards an effective framework for building
smart cities: lessons from Seoul and San Francisco. Technol. Forecast. Soc. 89,
80e99.
Li, X., Fong, P.S.W., Dai, S., Li, Y., 2019. Towards sustainable smart cities: an empirical
comparative assessment and development pattern optimization in China.
J. Clean. Prod. 215, 730e743.
Liao, S.J., Chen, X., Qian, Y., Shen, L.Y., 2017. Comparative analysis of the indicator
system for guiding smart city development. In: Proceedings of the 20th Inter-
national Symposium on Advancement of Construction Management and Real
Estate, pp. 575e594.
Lu, D., Tian, Y., Liu, V.Y., Zhang, Y., 2015. The performance of the smart cities in
China-a comparative study by means of self-organizing maps and social net-
works analysis. Sustain. Basel 7 (6), 7604e7621.
Manville, C., Cochrane, G., Cave, J., Millard, J., Pederson, J.K., Thaarup, R.K., Liebe, A.,
Wissner, M., Massink, R., Kotterink, B., 2014. Mapping Smart Cities in the EU.
Marsal-Llacuna, M.L., Colomer-Llinas, J., Melendez-Frigola, J., 2015. Lessons in urban
monitoring taken from sustainable and livable cities to better address the Smart
 A. Sharifi / Journal of Cleaner Production 233 (2019) 1269e1283
Cities initiative. Technol. Forecast. Soc. 90, 611e622.
Meijer, A., Bolivar, M.P.R., 2016. Governing the smart city: a review of the literature
on smart urban governance. Int. Rev. Adm. Sci. 82 (2), 392e408.
MMF, 2018. Global Power City Index 2018. Institute for Urban Strategies, The Mori
Memorial Foundation.
Mohan, A., Dubey, G., Ahmed, F., Sidhu, A., 2017. In: ISB), I.S.o.B. (Ed.), Smart Cities
Index: A Tool for Evaluating Cities.
Monfaredzadeh, T., Berardi, U., 2015. Beneath the smart city: dichotomy between
sustainability and competitiveness. Int. J. Sustain. Build. Technol. Urban Dev. 6
(3), 140e156.
Nam, T., Pardo, T.A., 2011. Conceptualizing smart city with dimensions of technol-
ogy, people, and institutions. In: Proceedings of the 12th Annual International
Digital Government Research Conference: Digital Government Innovation in
Challenging Times. ACM, pp. 282e291.
Naphade, M., Banavar, G., Harrison, C., Paraszczak, J., Morris, R., 2011. Smarter cities
and their innovation challenges. Computer 44 (6), 32e39.
NIST, 2018. IES-City Framework: A Consensus Framework for Smart City Architec-
tures. National Institute of Standards and Technology.
PWC, 2016. Cities of Opportunity 7. PricewaterhouseCoopers.
Rinaldi, S., 2017. The role of measurement in smart cities pilot projects [Future
I&M]. IEEE Instrum. Meas. Mag. 20 (6), 28e29.
Sang, Z., Ding, H., Higashi, M., Nakamura, J., Hara, M., Hashitani, T., Sugiura, J., Di
Carlo, C., Girdinio, P., Bolla, R., Gemma, P., Xu, D., Guo, J., Bergmark, P.,
Digeronimo, L., Castiella, L., 2015. Key Performance Indicators Definitions for
Smart Sustainable Cities. International Telecommunication Union.
SCA, 2014. Smart Cities Maturity Model and Self- Assessment Tool Guidance Note
for Completion of Self- Assessment Tool. The Scottish Government, Smart Cities
Alliance, Urban Tide.
SCBC, 2012. Smart Cities Benchmarking in China. China Academy of Telecommu-
nication Research of MIIT.
SCR, 2014. Smart Cities Maturity Model and Self–Assessment Tool Guidance Note
for Completion of Self-Assessment Tool. The Scottish Government and Scottish
1283
Cities Alliance.
Sharifi, A., 2016. A critical review of selected tools for assessing community resil-
ience. Ecol. Indicat. 69, 629e647.
Sharifi, A., Murayama, A., 2013. A critical review of seven selected neighborhood
sustainability assessment tools. Environ. Impact Assess. Rev. 38, 73e87.
Sharifi, A., Murayama, A., 2015. Viability of using global standards for neighbour-
hood sustainability assessment: insights from a comparative case study.
J. Environ. Plan. Manag. 58 (1), 1e23.
Sharifi, A., Yamagata, Y., 2016. Principles and criteria for assessing urban energy
resilience: a literature review. Renew. Sustain. Energy Rev. 60, 1654e1677.
Shen, L.Y., Huang, Z.H., Wong, S.W., Liao, S.J., Lou, Y.L., 2018. A holistic evaluation of
smart city performance in the context of China. J. Clean. Prod. 200, 667e679.
Simpson, P., 2017. Smart Cities: Understanding the Challenges and Opportunities.
Smart Cities World and Philips.
Stratigea, A., Papadopoulou, C.A., Panagiotopoulou, M., 2015. Tools and Technologies
for planning the development of smart cities. J. Urban Technol. 22 (2), 43e62.
Stratigea, A., Leka, A., Panagiotopoulou, M., 2017. In search of indicators for
assessing smart and sustainable cities and communities' performance. Int. J. E-
Plan. Res. 6 (1), 43e73.
Thielen, P., Hemis, H., Storch, A., Lutz, M., 2013. Gradual development of Austrian
smart city profiles. Blue global report. SmartCities 2, 2013.
Urbantide, 2016. Overview of the Smart Cities Maturity Model. Urban Tide.
Walter, C., Woodling, R., 2017. Smart City Domain Strategic Growth Map. ESPRESSO.
Woods, E., Omara, H., Ravens, S., Citron, R., 2016. Gulf States Smart Cities Index:
Assessment of Strategy and Execution for 10 Cities. Huawei.
Woods, E., Labastida, R.R., Citron, R., Chow, T., Leuschner, P., 2017. UK Smart Cities
Index 2017: Assessment of Strategy and Execution for the UK's Leading Smart
Cities. Navigant Research.
Wu, Z., Pan, Y., Ye, Q., Kong, L., 2016. The city intelligence quotient (city IQ) eval-
uation system: conception and evaluation. Engineering 2 (2), 196e211.
Zygiaris, S., 2012. Smart city reference model: assisting planners to conceptualize
the building of smart city innovation Ecosystems. J. Knowl. Econ. 4 (2), 217e231.