Article 1

‘Evaluation of software tools simulating landscape design ad- 2

aptation to climate change: an application of the ISO 25010 3

standard’ 4

Evelina Keibach1 and Homeira Shayesteh2 5

1 Middlesex University London 1; Evelina.Keibach@gmail.com 6

2 Middlesex University London 2; H.Shayesteh@mdx.ac.uk 7

Abstract: This paper investigates the capabilities and limitations of different software tools simulat- 8
ing landscape design adaptability. The evaluation of tools is based on the ISO 25010 framework, 9
which investigates software functionality, reliability, performance efficiency, usability, compatibil- 10
ity and information quality. These quality characteristics of software are analysed during objective 11
experiments where five software tools are used for the case study project at the conceptual design 12
phase. The experiments revealed that the existing software tools for climate adaptation planning are 13
focused on different aspects of climate adaptability, and generates different types of information. 14
Moreover, all tools deal with some limitations in terms of compatibility, performance efficiency and 15
functional operations. The ISO 25010 quality model provides a comprehensive framework to com- 16
pare the capabilities of different software tools for climate adaptation planning. 17

Keywords: BIM for climate adaptation, sustainable design, climate change, ISO 25010 standard; 18
19

1. Introduction 20
Climate change plays a significant role in the discipline of landscape architecture in 21
terms of climate adaptation. Climate change causes higher frequency and intensity of 22
floods and storms leading to increased financial losses [1]. According to the EU [2], the 23
Academic Editor: Homeira Shayes-
consequences of climate change caused damage to facilities counting €95 billion casualties 24
teh
between 2002 and 2012. The statistics predict that the EU will count €20 billion loss annu- 25
ally if the temperature increases by 2.5°C and €65 billion if it increases by 5.4°C [3]. There- 26
Received: 2021
fore, climate change challenges societies and planners to design adaptable cities efficiently 27
Accepted: date
and effectively to solve problems related to floods and heatwaves. However, adaptation 28
Published: date
to climate change planning deals with high complexity and requires innovative tools sup- 29

Publisher’s Note: MDPI stays neu-

tral with regard to jurisdictional
claims in published maps and institu-
tional affiliations.
Copyright: © 2021 by the authors.
Submitted for possible open access
publication under the terms and
conditions of the Creative Commons
Attribution (CC BY) license
(https://creativecommons.org/license
s/by/4.0/).
porting climate services. Currently, the possibilities of digital technologies for climate 30
adaption planning are not fully realised in practice. BIM (Building Information Model- 31
ling) concept based on data-rich models makes little or no contribution to landscape de- 32
sign with deep environmental concerns. One of the main barriers to the implementation 33
of climate adaptation planning tools is the lack of expertise and insufficient data to prove 34
the benefits of adaptive planning design [4]. Additionally, another issue is the lack of 35
knowledge of software tools supporting climate adaptation planning. However, the digi- 36
talisation of climate adaptive planning can improve efficiency in natural resources plan- 37
ning and support the decision-making process [5]. Moreover, it can help create innovative 38
solutions to the challenges caused by climate change, as climate adaptation software tools 39
can analyse climate conditions on-site, generate a large amount of data, calculate complex 40
climate variables, evaluate the impact of planning decisions on climate change and visu- 41
alise various scenarios. Furthermore, climate adaptation deals with a high level of uncer- 42

Appl. Sci. 2021, 11, x. https://doi.org/10.3390/xxxxx www.mdpi.com/journal/applsci

Appl. Sci. 2021, 11, x FOR PEER REVIEW 2 of 22

tainty when predicting future climate events. Digitalisation in climate adaptation plan- 43
ning would take a step forward towards data-based planning while reducing the risk of 44
failure and increasing planning efficiency. However, digital tools currently make little 45
contribution to the process of climate adaptation planning due to the fact that these tools 46
are rarely used in practice. Therefore, this paper seeks to evaluate the capabilities and 47
limitations of different climate adaptation software tools on the basis of the ISO 25010 48
framework. The research investigates the tools for climate adaptation planning during 49
objective experiments using a case study from Germany. Five different tools are tested on 50
a residential development project and their capabilities and limitations are compared us- 51
ing the ISO 25010 framework for product quality evaluation. The evaluation of these tools 52
leads to the identification of the most suitable software tool for the case study project. 53
Firstly, the paper will analyse the ISO 25010 software quality model which will be 54
adjusted for the evaluation of climate adaptation software tools for landscape conceptual 55
design with a focus on the environmental impact. Secondly, five different software tools 56
will be compared evaluating their functional suitability, reliability, performance effi- 57
ciency, operability, compatibility and information quality. Lastly, the research will sum- 58
marise the results from the objective experiments on the basis of the ISO 25010 framework 59
and will define the capabilities and limitations of software tools. The figure below presents 60
the structure of this paper. 61
62

ISO 25010 software quality model

Theoretical framework Adjustements

Analysis based on objective experiments

Functional Performance Compati- Information
Reliability Operability
suitability efficiency bility quality

Evaluation results

Summary of software evaluation Software strengths and weaknesses

63
Figure 1. The structure of the article. 64

65
66
Appl. Sci. 2021, 11, x FOR PEER REVIEW 3 of 22

2. Materials and Methods 67

68
The paper implements the inductive method to investigate different software tools 69
for climate adaptation projects. The inductive approach observes multiple facts without a 70
focus on pre-defined hypotheses. However, it does not prove the accuracy of observed 71
findings [6]. The paper seeks to find new and unexpected patterns between various soft- 72
ware tools. Therefore, the inductive method is chosen to conduct the research. 73
The research examines different software tools based on the ISO 25010 framework to 74
define the characteristics for evaluation. These are measured during objective experiments 75
with several software tools used in the case study. In order to observe different parameters 76
of the tools, qualitative research methods are introduced. Qualitative research methods 77
are mostly appropriate for experiments to find unexpected observations. The main differ- 78
ence between qualitative methods and quantitative methods is that qualitative methods 79
focus on the observed effect, while quantitative methods seek to measure the effect itself 80
[7]. This project-based research integrates descriptive analysis of the results using various 81
software tools. Table 1 presents the framework of the research showing how qualitative 82
methods will be applied to different subjects and identifies the type of method, subject of 83
evaluation, aims and tools. 84
85
Table 1. Methods, subjects of evaluation, aims and tools. 86
Subject of
Method Aim Method Tool
Evaluation
Functional Ergonomic quality
Qualitative observation Time register
Suitability
Qualitative Information quality Ergonomic quality observation description
Incidents
Qualitative Reliability Ergonomic quality observation and error
register
Performance
Qualitative Ergonomic quality observation Time register
efficiency
Qualitative Usability Ergonomic quality observation description
Qualitative Compatibility Ergonomic quality observation description
87
88
Moreover, various control levels define the types of experiments, which can be con- 89
trolled, observational and quasi-experiments [8]. Objective experiments with software 90
tools are rather observational due to the low level of control of variables, taking into con- 91
sideration human factors and the impact on results. However, the challenge of an objec- 92
tive study is the control of subjectivity when evaluating the capabilities of different soft- 93
ware tools. This research eliminates the subject of evaluation with the high risk of subjec- 94
tivity such as personal opinion and preferences. Therefore, the analysis of software tools 95
excludes some aspects of software ergonomics including user interface evaluation to elim- 96
inate subjective opinion and its effect on results. Objectivity is achieved by integrating 97
observational methods with a focus on the processing time of each task, the number of 98
errors and the final results of simulations. The variables on the time register such as the 99
operating system, computer capacity and parameters are eliminated by the use of one de- 100
vice. Nonetheless, simulations are calculated for the same time and date (19th of July) 101
using the same weather data from the Stuttgart area to achieve consistency for the com- 102
parative analysis. The experiments are performed using a conceptual design of a residen- 103
tial project with the same file as a base. The 3D model with the same layer structure and 104
elements was used to equally measure the compatibility and implementation process of 105
Appl. Sci. 2021, 11, x FOR PEER REVIEW 4 of 22

five climate adaptation software tools, including ENVI-met, Ladybug, GreenScenario, Cit- 106
ySim and AST (Adaptation Support Tool). These tools are selected due to their suitability 107
for landscape design projects focusing on climate adaptation. 108
The experiments with software tools were performed at several stages. Figure 2 illus- 109
trates the stages of experiments followed in this research. Firstly, a climate adaptation 110
project in the residential urban district within the outskirts of the city of Ulm was chosen 111
to evaluate its adaptability with software tools. The SketchUp model was prepared with 112
the use of a standard layer structure based on different categories of objects and materials. 113
The SketchUp software was chosen due to its suitability for the conceptual design phase 114
and popularity among landscape architects. Moreover, the analysis of the existing soft- 115
ware tools for climate adaptation planning was conducted to select the most suitable ones. 116
However, some software tools could not be included in the research due to low accessi- 117
bility. At the next stage, ISO 25010 software quality model was investigated and adjusted 118
for the evaluation of climate adaptation software tools. The next phase proceeded with 119
the testing of five different software tools to examine software evaluation aspects defined 120
in the adjusted ISO 25010 quality model. Lastly, the process and results were described 121
and compared. 122
123

•Project selection
•Model preparation for testing phase
1
•Software selection

•Adjustment of ISO 25010

2

•Testing ENVI-met
•Testing GreenScenario
•Testing Ladybug
3
•Testing CitySim
•Testing AST

•Describe the process & results

4
•Comparison of software tools
5
124
Figure 2. Stages of objective experiments. 125
126

3. Results 127

3.1. ISO 25010 standard 128

The evaluation of software quality integrates the ISO 25010 standard. It defines the 129
characteristics and sub-characteristics of quality in use model and product quality model 130
[9]. This research adopted the product quality model, which integrates the evaluation of 131
software reliability, functionality, usability, compatibility and performance efficiency. 132
Meanwhile, the quality in use model is focused on software usability and the interaction 133
with the user, and it evaluates effectiveness, efficiency, satisfaction, freedom of risk and 134
context coverage. Both quality models can be applied to climate adaptation software tools. 135
However, since this research seeks to achieve higher objectivity, it eliminates user satis- 136
faction analysis and focuses rather on the technical aspects of software tools. 137
138
139
Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 22

3.1.1. Adjustment of ISO 25010 for the evaluation of climate adaptation software 140
This section reflects how the ISO software product quality model is adjusted for the 141
climate adaptation software tools and how the essential quality aspects are integrated. 142
Figure 3 illustrates the changes of the original ISO quality model based on nine categories: 143
functionality, performance efficiency, reliability, security, operability, compatibility, 144
maintainability, and transferability [9]. The evaluation of climate adaptation software 145
tools excludes technical aspects relevant to software developers such as security, main- 146
tainability and portability. Functional suitability analyses the functionality of the software 147
including completeness, correctness and appropriateness. This study excludes the analy- 148
sis of functional correctness and focuses instead on the functionality compliant with sus- 149
tainability requirements such as BREEAM, LEED and DGNB. Moreover, the analysis of 150
software capacity and maturity is excluded due to the high complexity of measurements. 151
Moreover, some aspects such as appropriateness, recognisability and user interface aes- 152
thetics are excluded due to a high level of subjectivity. Some aspects, including learnabil- 153
ity and time behaviour, deal with a high level of individual capabilities but are important 154
guidelines that can be objectively evaluated, 155
Moreover, the ISO standard does not consider information quality as an important 156
aspect of software quality. However, information quality is the core of BIM projects, hence 157
it is essential to add this aspect to the quality requirements. Information quality can be 158
assessed in terms of accuracy, suitability, accessibility and legal compliance [10]. For this 159
case study, information quality will be evaluated based on the accuracy of simulations or 160
calculations and suitability for climate adaptation planning. 161

Compliance with
sustainability requirements

162
163
Figure 3. ISO 25010 product quality model adjusted for climate adaptation project [9]. 164
165
3.1.2. Software evaluation framework for climate adaptation simulations 166
Table 2 summarises the main quality measures of climate adaptation software tools 167
and the main targets measured during an objective experiment. The description of each 168
aspect is based on the definitions provided in ISO 25010 while the targets define the type 169
of measures for the evaluation of different software tools. Further on, the comparison of 170
software tools describes each quality measure. 171
172
173
Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 22

Table 2. Software evaluation framework for climate adaptation software based on ISO 174
25010 standard 175

ISO Requirements Description Targets

Functional Suitability
Functional The accomplishment of the task excludes unnecessary steps. Evaluation of process maps
appropriateness
Functional The analysis of various climate adaptation aspects is covered Evaluation of
completeness in the software. software functions
Compliance with The analysis on the sustainability aspects covered. Evaluation of
sustainability software compliance with
certifications sustainability standards
Reliability

Software Availability Evaluation of the software reliability Calculation of incidents

Fault tolerance
Recoverability
Performance efficiency
software ‘not responding’
Evaluation of the software operation despite hardware or Calculation of errors and
software faults. failures
The software can recover the data in case of interruption Evaluation of recovered data in
case of system error

Time Behavior The time needed for task processing and simulations. Calculation on time used per
task
Resource utilization The resources needed for tasks (excludes the productivity) Evaluation of the resources
(expert consultations, training
material, other files and
software tools) needed to
perform the task
Usability
Learnability The time to learn to use the software for climate adaptation Calculation on time for training
functions. for a specific task
Operability The software can be easily operated then learned. Evaluation of the software
complexity
User error protection The software can help users to avoid errors. Evaluation of error
identification.
Compatibility

co-existence The software performs efficiently, sharing a common Evaluation of how the software
environment and resources with other software tools is compatible with other
software
Interoperability The software can exchange and use the information from other Evaluation on the data lost or
software geometrical issues in the
exportation and importation
Information Quality
Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 22

Accuracy How accurate are the simulations or calculations Evaluation of the accuracy of
simulations
Suitability How informs on the climate adaptation performance Evaluation of how the results
inform on climate adaptation
planning
176
3.2. Functional suitability 177
Functional suitability focuses on the functional appropriateness of software for cli- 178
mate adaptation projects, functional completeness and compliance with sustainability cer- 179
tifications. This section explains how these parameters are evaluated and measured. 180
181
3.2.2. Functional completeness 182
Functional completeness analyses the main functions of software tools in terms of the 183
tasks and objectives of climate adaptability. The experiment with different software tools 184
revealed a range of climate adaptation parameters in different software tools. Table 3 gives 185
an overview of the functional completeness and defines the measures of different software 186
tools based on several categories including green and blue measures, climate analysis and 187
economic impact. Software tools tested in the objective experiments revealed a different 188
focus on climate adaptation. For instance, Envimet and Ladybug can simulate a wide 189
range of climate aspects including sun, temperature, radiation, wind and thermal comfort, 190
whereas GreenScenario and AST evaluate green and blue measures as well as economic 191
impact. CitySim covers the least of climate adaptability aspects for outdoor spaces pro- 192
ducing simulations of shading areas, surface temperature and radiation. 193
194
Table 3. Comparison of functional completeness of different software tools. 195
Parameter ENVI- Lady- City- Green- AST
Met bug Sim Scenario
GREEN MEASURES
Green factor +
Biodiversity +
Shading area + + +
BLUE MEASURES
Rainwater management + +
Evapotranspiration + + +
CLIMATE ANALYSIS
Solar analysis + +
Surface Temperature + +
Radiation + + +
Air pollution +
Wind analysis + +
Thermal comfort + +
Thermal load +
Thermal Storage +
Heat island reduction + + +
CO2 reduction + +
Appl. Sci. 2021, 11, x FOR PEER REVIEW 8 of 22

ECONOMIC IMPACT
Maintenance costs + +
Construction costs + +
11p. 6p. 3p. 8p. 5p.
196
197
3.2.1. Functional appropriateness 198
Firstly, functional appropriateness according to ISO 25010 is focused on the efficient 199
accomplishment of the task without taking unnecessary steps. Therefore, functional ap- 200
propriateness evaluates the number of steps used to perform the task. On the contrary, 201
the duration of each task is separately assigned to performance efficiency, as the ISO qual- 202
ity model proposes. Table 4 summarises the results of objective experiments using the 203
information from the process maps and time register assessing functional appropriateness 204
and performance efficiency. The tasks to perform simulations can be divided into the main 205
phases including importation of the SketchUp model used for experiments, adjustments 206
of model or remodelling, settings to perform simulations, simulating time and results. 207
The results show that the most efficient software tools were AST, Greenscenario and 208
CitySim. However, the results revealed that CitySim was very inefficient during the sim- 209
ulation process, which was longer than 98 hours. Ladybug is based on the visual script 210
used to perform simulations, however, a new script is needed for a different type of anal- 211
ysis. Most steps were used for Envimet software, which is more complex than the others 212
and allows to create many microclimate simulations. The results show that the compati- 213
bility of the software with SketchUp has a great impact on the final results of functional 214
appropriateness. 215
216
Table 4. The comparison of functional appropriateness and performance efficiency. 217
Envimet LadyBug AST Greenscenario CitySim Pro

Importation 1 Model Preparation 1 Import SketchUp -No compatibility 1 Clean 3D model 1 Export to AutoCAD;

Compatibility using SketchUp INX to Rhino Duration: 20mins Duration: 2mins

Duration: 1h30min Duration: 2mins 2 Export to AutoCAD 2 Prepare DXF model

Duration: 3mins Duration: 5mins

2 Exportation to 3 Export dwg to Rhino 3 Import DXF into

ENVI-met Spaces Duration: 2mins SIM

Duration: 5mins Duration: 2mins

3 Solving errors

Duration: 2h

Model 4 Adjustments in 1 Assign the project 4 Extrude the buildings

adjustments/ ENVI-met location and and trees;

remodelling Spaces/Albero boundary; Duration: 1h 15mins

Duration: 1h Duration: 5mins 5 Assign materials to

2 Define the surfaces;

characteristics of the Duration: 2h 20mins

area;

Duration: 2mins

3 Define climate

adaptation goals;
Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 22

Duration: 5mins

4 Choose climate

adaptation measures;

Duration: 2mins

5 Draw the measures

on the map;

Duration: 2h

6 Generate the final

design;

Duration: 10mins

Settings 5 Setting up the 2 Setting the - 6 Prepare visual script; 4 Prepare cli. and hor.

simulation Simulation 1 Duration: 15mins Files;

Duration: 5mins (visual scripting

method) 5 Set up the attributes

Duration: 45mins of materials; Duration:

3 Setting the 5mins

Simulation 2

Duration: 30mins

4 Setting the

Simulation 3

Duration: 30mins

Simulations 6 Simulating 5 Simulating - 6 Simulation

Duration: 21h Simulation 1 Duration: 98h

Duration: 2mins

6 Simulating

Simulation 2

Duration: 2mins

7 Simulating

Simulation 3

Duration: 2mins

Results 7 Illustration of -

simulation results

Duration: 30min

8 Running BIOMET - -

simulations

(additional)

Duration: 1h20min

Overall Duration: Overall Duration: Overall Duration: Overall Duration: Overall Duration:

27h30min 2h23min 2h24min 4h12min 98h14min

Steps: 8 Steps: 7 (depends Steps: 6 Steps: 6 Steps: 6

on the number of

simulations)
Appl. Sci. 2021, 11, x FOR PEER REVIEW 10 of 22

3.2.3. Compliance with sustainability requirements 218

The requirements of different sustainability programs reveal slight interpretations of 219
climate adaptation goals. DGNB excludes climate change analysis and economic value, 220
while BREEAM and LEED include ecology and green infrastructure requirements. Table 221
5 summarises the analysis of sustainability requirements of different sustainability pro- 222
grams. The results show that Greenscenario and AST tools consider the most of aspects. 223
However, this evaluation excludes detailed research on each sustainability criteria and 224
how well each software can perform analysis of each aspect. 225
226
Table 5. Compliance with sustainability requirements of different software tools. 227
Requirement ENVI- lady- City-sim Green- AST
Met bug Scenario
BREEAM
Flood Risk Assessment & +
Management
Rainwater analysis + +
Microclimate Analysis + + +
Green Infrastructure analysis + + +
Ecological value analysis + +
LEED
Floodplain analysis + +
Rainwater analysis + +
Green Infrastructure analysis + + +
Impact on Heat Island + + +
Ecological analysis (Habitats) + +
DGNB
Climate change analysis +
Rainwater Management + +
Water & Soil analysis + +
Life cycle costs
Environmental impact + + + + +
5p. 3p. 2p. 10p. 13p.
228
229
3.3. Information quality 230
Functional suitability for climate adaptation projects analysed in the previous chap- 231
ter shows high variations within the functionalities and focus on climate adaptation. This 232
chapter further investigates the information gathered from different software tools, which 233
can be divided into visual and digital information. ENVI-met and Ladybug generate vis- 234
ual information through simulations, while Greenscenario and AST produce calculations 235
based on climate adaptation measures. Information quality evaluates the accuracy of in- 236
formation and suitability for climate adaptation planning. 237
238
Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 22

3.3.1.Visual information 239

The comparison of simulations of sun hours, radiation and UTCI of Ladybug and 240
Envimet show significant differences between the simulated values. For instance, Lady- 241
bug indicates the lowest radiation of 109 W/m2 whereas ENVI-met indicated only 69.4 242
W/m2 (see Figure 4). These variations might be a result of different calculation methods, 243
as the results of Ladybug are based on the visual scripting method developed by the user, 244
and ENVI-met has a built-in formula. However, the results can be compared visually 245
when evaluating sun hours and shaded areas (Figure 4) and the most radiated areas (see 246
Figure 5). Both programs show the lowest radiation levels in the shaded areas around 247
buildings and trees. CitySim can produce visual information on radiation and sun hours, 248
however, the accuracy of this information depends on the triangulation of the surfaces 249
and the model. In the objective experiments, CitySim generates the least informative sim- 250
ulations regarding climate adaptation planning. Furthermore, Figure 6 illustrates UTCI 251
values that are of high complexity. Therefore, Envimet can produce more accurate simu- 252
lations than Ladybug. Moreover, in other papers, Envimet is introduced as accurate and 253
trustable software often used to verify visual scripts in Ladybug [11]. 254

255
(a) (b)
256
Figure 4. Comparison of the simulations of sun hours: (a) ENVI-met; (b) Ladybug; 257
258
Appl. Sci. 2021, 11, x FOR PEER REVIEW 12 of 22

259
(a) (b)
260
Figure 5. Comparison of radiation simulations: (a) ENVI-met; (b) Ladybug; 261
262

263
(a) (b)
264
Figure 6. Comparison of UTCI simulations: (a) ENVI-met; (b) Ladybug; 265
266
3.3.1.Digital information 267
AST and Greenscenario present digital results on climate adaptation performance. 268
Both software tools provide calculations of the impact of climate adaptation measures on 269
climate by estimating rainwater management, evapotranspiration, heat reduction and 270
costs. Greenscenario has an advantage over AST as it presents information on a visually 271
attractive dashboard (see Figure 7). Moreover, the calculations present the impact of the 272
Appl. Sci. 2021, 11, x FOR PEER REVIEW 13 of 22

whole design, while AST gives an overview of the impact of each climate adaptation 273
measure (see Figure 8). Due to different types of calculations and values, the results are 274
hardly comparable. The comparison of costs for construction and maintenance shows a 275
significant difference: Greenscenario calculates 125 e/m2 for construction and 12.3 e/m2*y 276
for maintenance, while AST calculates 27.5e/m2 and 2 e/m2*y, respectively. This differ- 277
ence might occur due to different climate adaptation measures assessed in both tools. 278
Moreover, AST deals with a high level of inaccuracy in the drawings. Therefore, Greensce- 279
nario can produce more accurate assessments allowing to compare different climate ad- 280
aptation goals in a visually attractive manner. 281
282

283
Figure 7. The final climate adaptation plan using Greensceario. 284
285
Appl. Sci. 2021, 11, x FOR PEER REVIEW 14 of 22

286
Figure 8. The impact on climate and costs calculated with AST. 287
288
3.4. Reliability 289
Reliability of different software tools includes software availability, fault tolerance 290
and recoverability. Software availability evaluates the accessibility of software ensuring 291
its consistent functioning without incidents. To evaluate software availability, the inci- 292
dents of ‘not responding’ are calculated from the error register. Moreover, the connection 293
and software stability is assessed. The only software that encountered the ‘not respond- 294
ing’ issue during the experiment was Ladybug. The issue occurred when setting high- 295
quality simulations. Another software with low stability is AST, which relies on an inter- 296
net connection. CitySim is lagging due to the large number of calculations required to 297
perform simulations. Other software tools are rather stable, as no incidents were recorded 298
during experiments. 299
Fault tolerance considers errors and failures as the main aspect indicating software 300
faults. The hardest faults were faced using CitySim since the file could not be reopened 301
due to errors in the model. ENVI-met application for SketchUp also faced issues while 302
reopening the files leading to information and data loss. Ladybug and Greenscenario 303
based on the visual script method identified some errors in the scripts. Even though AST 304
was not facing any errors or failures, it does not integrate the function helping to identify 305
errors. 306
Recoverability analyses if the software can recover data in case of interruption. AST 307
software does not integrate the function allowing to save the project, therefore data recov- 308
ery is not possible. CitySim has revealed issues while reopening the files. Envi-met had 309
no failures or issues with data loss, however, the application for SketchUp could not re- 310
cover information after reopening the file. Greenscenario and Ladybug, which are based 311
on the Rhino interface, can recover files in case of failure. 312
313
3.5. Performance efficiency 314
Performance efficiency includes time behaviour and resource utilisation according to 315
ISO 25010. The evaluation of on-time behaviour is based on the time register illustrated in 316
Table 4, which provides information on time resources used for an objective experiment 317
per task. The results revealed that overall time resources were the highest for CitySim and 318
Appl. Sci. 2021, 11, x FOR PEER REVIEW 15 of 22

the lowest for Ladybug and AST. Moreover, CitySim required the most resources to pro- 319
ceed with simulations such as expert consultations. The majority of software tools require 320
additional files with climate information such as .epw files for Ladybug and ENVi-met, 321
.dly for Greenscenario and .cli and .hor for CitySim. The least resources were used for the 322
AST software as well as training materials. 323
324
3.6. Usability 325
Usability includes the analysis of learnability, operability and user error protection. 326
Learnability defines the ability to learn the software, which can be easily operated 327
after learned. The evaluation of learnability is based on the time register marking the time 328
for training. The easiest software to learn was AST due to its simplicity and clarity. Further 329
on, CitySim also has a simple interface requiring less time than ENVI-met. The most com- 330
plicated software tools were Greenscenario and Ladybug as they require knowledge of 331
visual scripting. 332
Operability evaluates how the software operates after the learning phase. The most 333
complex software tool is ENVI-met as it integrates several applications and complex set- 334
tings and requires knowledge of climatology. Similarly, Ladybug requires knowledge of 335
climate calculation formulas, which has a high impact on results. Greenscenario has an 336
integrated formula to calculate the results from the model. CitySim and AST software 337
tools can be distinguished as the easiest to use. 338
User error protection evaluates software protection against errors by employing the 339
model checker function. ENVI-met integrates the model checking function comprehen- 340
sively to analyse the failures in the model before the simulation is started. CitySim scans 341
for the potential model failures when reopening the file. Ladybug and Greenscenario have 342
an automatic error finder for the failures in the script but not in the model. Meanwhile, 343
AST does not contain error protection measures. 344
345
3.7. Compatibility 346
Compatibility evaluates co-existence with other software tools and interoperability. 347
Interoperability evaluates the importation process of the model using the results from Ta- 348
ble 4. In this case, objective experiments were performed using the same 3D SketchUp 349
model. The results revealed that each software tool requires different types of files and 350
models for a successful model importation. 351
Firstly, Ladybug and Greenscenario are mostly compatible as both are based on the 352
Rhino interface which is compatible with a number of other planning software tools from 353
Autodesk to Graphisoft. However, Greenscenario requires a 2D file with closed and 354
grouped polylines which are extruded when assigning information of materials, levels of 355
each building and vegetation. Further on, CitySim is compatible with AutoCAD software, 356
requiring the model compounded from 3d faces. In this case, the SketchUp model was 357
exported to AutoCAD where the faces were converted to 3d faces and saved as dxf. En- 358
vimet is the only software suggesting a plugging for SketchUp, which allows preparing 359
the files for exportation. However, the experiment revealed some inefficiencies causing 360
errors and information loss leading to the necessity of remodelling. AST tool fails to inte- 361
grate the SketchUp model and is rather designed for renovation projects than for new 362
developments. Therefore, the design had to be redrawn due to failure to achieve precision. 363
364
3.8. Overview of evaluation results 365
The results from the objective experiments revealed limitations of each software tool. 366
The evaluation of software quality according to ISO 25010 uses five grade system to rank 367
software tools. Table 6 gives an overview of software tools that meet the most of the ISO 368
requirements. The scores are given in a comparative principle ranking five software tools 369
Appl. Sci. 2021, 11, x FOR PEER REVIEW 16 of 22

from the highest score 5 to the lowest score 1. The highest score was given to Greensce- 370
nario tool and the lowest score was given to CitySim. However, there is a small difference 371
between Ladybug, AST and ENVI-met. Nonetheless, overall scores do not reflect the ca- 372
pabilities and limitations of each software tool. Therefore, Figure 9 presents a comparative 373
analysis of the capabilities and limitations of software tools based on the ISO 25010 frame- 374
work. The results reveal that functional suitability is the strength of Greenscenario and 375
AST, while ENVImet is the strongest in information quality. Despite the strength in per- 376
formance and operability, ENVI-met and AST reveal some limitations in compatibility 377
and reliability. Due to its simplicity, AST can be efficiently used for climate adaptation 378
scenarios, however, the results on information are rather limited. To conclude, the results 379
show that the time invested to produce simulations ensures better information quality. 380
381
Table 6. Software evaluation based on ISO 25010 382
Quality characteristics ENVI- Green- Ladybug CitySim AST
and Sub-characteristics met Scenario Pro
1. Functional suitability
- Functional appropriateness 1 4 2 3 5
Functional completeness 5 4 3 1 2
- Compliance with sustainability 3 4 2 1 5
requirements
2. Reliability
- Software availability 5 5 1 3 2
- Fault tolerance 2 3 3 1 5
- Recoverability 3 5 5 2 1
3. Performance Efficiency
- Time behaviour 2 3 5 1 4
- Resource utilization 2 3 4 1 5
4. Usability
- Learnability 3 1 1 4 5
- Ease of use 1 3 2 4 5
User error protection 5 3 3 4 1
5. Compatibility
- Co-existence 2 5 5 4 1
- Interoperability 2 4 5 3 1
6. Information Quality
- Accuracy 5 3 4 1 2
- Suitability 5 4 2 1 3
OVERALL 46 54 47 34 47
383
384
Appl. Sci. 2021, 11, x FOR PEER REVIEW 17 of 22

Software Comparison
Functionality

Information Quality Reliability

Compatibility Performance

Usability

ENVI-met Ladybug CitySim AST GreenScenario

385
Figure 9. The comparison of software tools concerning different ISO 25010 aspects. 386
387
3.9. Limitations of results 388
The results of the research are limited to the specific project and model. Moreover, 389
the research investigates the performance of software tools using a conceptual design 390
phase. The choice of software tools is limited on the tools supporting outdoor simulations 391
or calculations excluding mono-functional and elementary software tools. Additionally, 392
the research includes only accessible software tools and the tools which were shared for 393
this research. 394
The analysis of the compliance with the requirements of sustainability certifications 395
such as LEED, BREEAM, DGNB excludes a broad examination of how well the software 396
can cover each aspect of sustainability. The results are rather constrained on the sustaina- 397
bility categories defined in different certifications, analysing the capability of the software 398
to provide the results suitable for each category. 399
Nonetheless, the experiments deal with a high level of objectivity as they are per- 400
formed by a single user. Therefore, some ISO 25010 quality aspects are excluded due to a 401
high impact of objectivity regarding personal preferences and choices. 402

4. Discussion 403
4.1. ISO quality model evaluation for software tools 404
This research is focused on the evaluation of climate adaptation software tools for 405
landscape projects based on the ISO 25010 framework. Integration of the ISO framework 406
provided a comprehensive and structured guideline for software evaluation. The most 407
important aspects such as software functionality, reliability, performance, usability and 408
compatibility are evaluated using the definitions from ISO and guidelines. However, one 409
of the most important aspects in data-rich models – information quality – is missing in the 410
ISO framework [10]. The paper analysed information quality by comparing the final out- 411
put produced with different software tools in the same case study project. This compara- 412
tive analysis identified different types of information including visual and digital out- 413
comes produced with different software tools. The literature review presented that the 414
majority of papers focus on a single simulation tool and one climate adaptation measure 415
such as green roofs [12], vegetation arrangement [13-17] and materials [18]. This paper 416
Appl. Sci. 2021, 11, x FOR PEER REVIEW 18 of 22

gives an overview of the simulation processes and the final results of different tools. None- 417
theless, the paper compares the performance efficiency of different software tools with 418
regard to time resources used for the tasks to perform simulations. The results show that 419
high-quality simulations often require more time resources for simulation production and 420
time to learn the software. 421
The ISO framework allows to compare the capabilities of different software tools 422
using a standard structure covering the most important aspects for software users. How- 423
ever, the ISO framework is rarely used for the comparison of climate adaptation software 424
tools in other papers as they cover different aspects. For instance, Vidmar and Roset [19] 425
focused on functionality, usability, user interface and ease of use. According to the ISO 426
framework, this paper covers only two categories: functionality and usability. Extensive 427
research on thermal comfort outdoors compared eight different software tools seeking to 428
find the best tool for outdoor performance simulations [20]. The paper compares general 429
software features including user interface, software reliability, information accuracy, com- 430
patibility, graphics and comfort prediction index. ISO defines user interface as a part of 431
software usability analysis. Moreover, this paper excludes the evaluation of the user in- 432
terface to avoid subjectivity in the evaluation phase as it depends on personal preferences. 433
Other papers add other essential aspects for consideration that are not included in the ISO 434
framework, such as the price of software and the flexibility to adjust to individual projects 435
[21]. The price and software flexibility are often the main criteria for software selection in 436
individual companies. Nonetheless, software flexibility is highly influenced by software 437
functionality and features. 438
Functional suitability, information quality and performance efficiency are the most 439
critical aspects when choosing software since functional suitability represents the adjust- 440
ability of tools to the climate adaptation projects allowing to estimate the value created 441
for climate adaptation projects. Moreover, the comparison of software output gives an 442
overview of software capabilities and functionality. Other aspects such as software com- 443
patibility, reliability and usability have a direct impact on software performance. Software 444
reliability and compatibility can improve performance efficiency while reducing the risk 445
of rework. Nonetheless, the ISO framework has a broad applicability side which can be 446
adapted to individual projects choosing the most important aspects for analysis. How- 447
ever, most of the papers are limited to the analysis of simulations with a focus on partic- 448
ular software features [20, 22-24]. Other papers compared the simulations between differ- 449
ent software tools to validate the results [23] and to define the differences between the 450
calculation rules [24]. This paper seeks to analyse the simulations performed with differ- 451
ent software tools for the same climate adaptation project to compare software capabili- 452
ties. 453
454
4.2. Challenges of the implementation of climate adaptation software 455
Currently, the integration of software tools simulating climate adaptation design is 456
rather uncommon. There is a number of barriers reducing the adaptability of climate ad- 457
aptation software tools for landscape projects. Firstly, it can cause additional costs for soft- 458
ware and for the time spent to proceed with simulations. Therefore, the performance effi- 459
ciency of these tools plays a significant role. The analysis shows that precise simulations 460
require time for training, modelling and simulating. Additionally, low compatibility with 461
planning tools cause remodelling and rework. However, the issues regarding software 462
performance have not been well researched in the literature so far as it is more focused on 463
other aspects such as user interface or graphical information. Moreover, the paper identi- 464
fied different functional abilities, as each software tool measures different climate adapta- 465
tion parameters. This might lead to the need to combine the information from different 466
software tools to cover the main aspects of climate adaptation planning. In practice, plan- 467
ners are seeking to find one tool covering diverse aspects with the potential to adapt to 468
different projects [21]. 469
Appl. Sci. 2021, 11, x FOR PEER REVIEW 19 of 22

However, the value of software tools for climate adaptation planning has not been 470
widely researched so far. One research that analyses the effect of tools on planning com- 471
pares the design results using tools and a tools-free approach in the workshop based on 472
collaborative planning [25]. The results revealed that the tools used in the experimental 473
workshop had a significant impact on the learning process, collaboration and communi- 474
cation, and the final planning results. The simulations in this research shows how the 475
planning process can be improved using these tools. For instance, the simulation of the 476
temperature revealed how different arrangements of vegetation and buildings could sig- 477
nificantly reduce heat island effects. Currently, most climate adaptation designs are fo- 478
cused on the reduction of temperature in urban areas. However, climate adaptation has 479
more measures, including rainwater management, ecological quality and economic effect 480
on the community [26]. The main criteria for climate adaptation planning can be reflected 481
in software tools. The research revealed that climate adaptation software tools focus on a 482
different aspect of climate adaptation. Some tools include the analysis of the environment, 483
but only a few consider the ecological and social benefits of climate adaptation planning. 484
One of the main problems is the lack of standardisation of climate adaptation measures, 485
as sustainability certifications such as LEED, DGNB and BREEAM identify different di- 486
mensions for climate adaptation planning. 487
4.3. Future research 488
Further study on climate adaptation software tools implementing more experiments 489
with different planners could help to verify the results of this paper as this research is 490
limited to the performance of experiments by one person. Therefore, the comparison of 491
the results of different persons could help achieve higher objectivity. This research re- 492
moves the evaluation of some quality aspects from the ISO framework due to a high risk 493
of subjectivity. The elimination of personal preferences and choices can be better achieved 494
if experiments are performed with more individuals. 495
Moreover, other research could investigate the necessity to implement climate adap- 496
tation software tools in different planning phases. Interviews with planners and experi- 497
ments would help to compare the impact of software tools on the planning process and 498
final design. The analysis could define the influence that software tools have on the choice 499
of design scenarios. Moreover, the research investigating the impact of software tools on 500
the dialogue with clients would answer the question how different visualisations of cli- 501
mate adaptation information influence the choices of design concepts. It would help to 502
identify the extent of visual informativeness of the simulations of climate adaptation plan- 503
ning for a different types of clients. 504
505
4.4. SWOT 506
SWOT method was integrated to define the strengths, weaknesses, opportunities and 507
threats of this paper. A summary of the results is reflected in Table 7. 508
Firstly, one of the strengths of the paper is the adaptation of the ISO 25010 framework 509
to software quality evaluation by integrating standardised procedures. The majority of 510
other papers analyse software tools using different aspects so that integrity and compre- 511
hensiveness are lost. Moreover, the objective experiments proceeded by one person help 512
to reduce the variables such as diverse operational systems and device properties. The 513
paper analyses five different software tools showing the process and different outputs 514
using the same project. 515
The weaknesses of the paper include the level of individualisation and specific con- 516
text. Firstly, software tools are used for a specific project without consideration of wider 517
applicability to different climate adaptation projects. Moreover, the results, especially 518
learnability and performance efficiency, are highly influenced by the capabilities of an 519
individual person. Nonetheless, the objective experiments in this research were per- 520
formed without any software knowledge of the tested tools. However, some tools were 521
not included in the research due to low public accessibility. 522
Appl. Sci. 2021, 11, x FOR PEER REVIEW 20 of 22

The paper informs software developers on the demand for climate adaptation needs 523
and issues within the current software tools and creates the potential for improvement. 524
The revision of the limitation of climate adaptation planning tools can lead to an improved 525
planning process and informed decisions by planners and clients. 526
One of the main threats to the implementation of climate adaptation projects on a 527
larger scale is a different understanding of climate adaptation concepts, measures and 528
impact. Moreover, low awareness about software tools and the need for digitalisation of 529
climate adaptation planning lead to low integration of software tools. 530
531
Table 7. SWOT Analysis 532
Helpful Harmful
Strengths Weaknesses
Implementation of ISO framework to Software tools were evaluated based on
the research helped to standardize the findings of one project;
software evaluation process; The research excludes the
Internal origin

Objective experiments proceeded by considerations on wider software

one person on the same device applicability;
reduced the level of variables; The research findings are partly
The most suitable tools for climate influenced on the individual
adaptation projects were tested by capabilities;
comparing the output of different Some software tools could not be tested
tools and processes. due to low public accessibility;
Opportunities Threats
The development of software tools can Different understanding of climate
External origin

meet climate adaptation planning adaptation;

needs; A low number of planners pursuing
Software tools can improve the climate adaptation projects;
planning process for planners based Low integration of software tools in
on informed decisions; climate adaptation planning process;

533

5. Conclusions 534
The paper analyses software tools for climate adaptation planning based on the ISO 535
25010 software quality model. The objective experiments were performed to analyse the 536
capabilities and limitations of different tools and to identify the best tool for the case study 537
project. The ISO framework helped to define the characteristics for the analysis of software 538
tools. The paper identified the limitations of the tools and the main differences between 539
them. The most common issues are concerned with software interoperability, data loss, 540
functionality and low interactivity. Low interoperability with the planning tools requires 541
the adjustments of models to perform experiments or remodelling. Moreover, the results 542
of simulations showed that different tools cover different functions of climate adaptation 543
planning. Therefore, the combination of these tools can bring more value to the project. 544
545
Appl. Sci. 2021, 11, x FOR PEER REVIEW 21 of 22

Author Contributions: Conceptualization, E.K. and H.S.; methodology, E.K.; formal analysis, E.K.; 546
investigation, E.K..; resources, E.K..; data curation, H.S., writing—original draft preparation, E.K.; 547
writing—review and editing, H.S.; visualization, E.K.; supervision, H.S. All authors have read and 548
agreed to the published version of the manuscript.” 549

Funding: This research received no external funding 550

Institutional Review Board Statement: Not applicable. 551

Informed Consent Statement: Not applicable. 552

553

Acknowledgments: The authors are thankful for the software companies sharing the materials of 554
the software tools used for experiments including ENVI-met, Greenscenario and CitySim. 555

Conflicts of Interest: The authors declare no conflict of interest. 556

557

References 558
559
1. ‘Explainer: Are natural disasters on the rise?’. Available online: http://theconversation.com/explainer-are-natural-disasters-on- 560
the-rise-39232 (Accessed on 07 November 2021). 561
2. ‘Towards an EU Research and Innovation policy agenda for Nature-Based Solutions & Re-Naturing Cities Final Report of the 562
Horizon 2020 Expert Group on ‘Nature-Based Solutions and Re-Naturing Cities’. Available online: https://op.eu- 563
ropa.eu/en/publication-detail/-/publication/fb117980-d5aa-46df-8edc-af367cddc202 (Accessed on 07 November 2021). DOI: 564
10.2777/765301. 565
3. Ciscar, J.C., Iglesias, A., Feyen, L., Szabó, L., Van Regemorter, D., Amelung, B., Nicholls, R., Watkiss, P., Christensen, O.B., 566
Dankers, R., Garrote, L., Goodess, C. M., Hunt, A., Moreno, A., Richards, J., Soria, A. (2011). ‘Physical and economic conse- 567
quences of climate change in Europe’. Proceedings of the National Academy of Sciences, Potsdam Institute for Climate Impact 568
Research, Potsdam, Germany. Editor Schellnhuber, H. J. DOI: 10.1073/1011612108. 569
4. Ernst, C., Blaha, K. (2016). ‘Decision support tools for climate change planning’. The Trust For Public Land’s Climate-Smart 570
Cities program. Available online: https://www.tpl.org/decision-support-tools-climate-change-planning (Accessed on 07 No- 571
vember 2021). 572
5. Balogun, A. L., Marks, D., Sharma, R., Shekhar, H., Balmes, C., Maheng, D., Arshad, A., Salehi, P. (2020). ‘Assessing the Poten- 573
tials of Digitalization as a Tool for Climate Change Adaptation and Sustainable Development in Urban Centres’. Sustainable 574
Cities and Society, Volume 53, February 2020. DOI: 10.1016/2019.101888 575
6. Lodico, M.G., Spaulding, D.T, Voegtle, K.H. (2010). ‘Types of Approaches Used in Educational Research’. In Methods in Educa- 576
tional Research: From Theory to Practice, 1st ed., Editor Publisher: Jossey-Bass, San Francisco, Unites States of America, 2006. Vol- 577
ume 1, pp. 10–21. 578
7. Hasselbring, W., Giesecke, S. ‘Integration of Qualitative and Quantitative Methods’. In Research Methods in Software Engineering, 579
2006. Volume 1, pp. 61–81. 580
8. Easterbrook, S., Singer, S., Damian, D. ‘Selecting Empirical Methods for Software Engineering Research’. In Guide to Advanced 581
Empirical Software Engineering, Shull F., Singer J., Sjøberg D.I.K. (eds) Springer, London, 2008. DOI: 10.1007/978-1-84800-044- 582
5_11. 583
9. BS ISO/IEC 25010:2011. ‘Systems and software engineering —Systems and software Quality Requirements and Evaluation 584
(SQuaRE) — System and software quality models’. 585
10. Lew, P., Olsina, L., Zhang, L. ‘Quality, Quality in Use, Actual Usability and User Experience as Key Drivers for Web Application 586
Evaluation’. Conference: Web Engineering, 10th International Conference, ICWE 2010, Vienna, Austria, 2010. Benatallah B., 587
Casati F., Kappel G., Rossi G. (Eds.), LNCS 6189, Springer, pp. 218-232. DOI: 10.1007/978-3-642-13911-6_15 588
11. Ibrahim, Y., Kershaw, T., Shepherd, P. ‘Improvement of the Ladybug-tools microclimate workflow: A verification study’. Con- 589
ference: Building Simulation and Optimization, Loughborough University, UK, 2020. 590
12. Kraus, F., Scharf, B. ‘Green Roofs and Greenpass’. In Buildings 2019, 9(9), 205. DOI: 10.3390/buildings9090205. 591
13. Chatzinikolaou, E., Chalkias, C., Dimopoulou, E. ‘Urban microclimate improvement using ENVI-Met climate model’. The In- 592
ternational Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4, 2018. ISPRS 593
TC IV Mid-term Symposium “3D Spatial Information Science – The Engine of Change”, 1–5 October 2018, Delft, The Nether- 594
lands, 2018. DOI: 10.5194/isprs-archives-XLII-4-69-2018. 595
14. Zheng, S., Zhao, L., Li, Q. ‚ Numerical simulation of the impact of different vegetation species on the outdoor thermal environ- 596
ment’. In Urban Forestry & Urban Greening, 2016, Vol. 18, pp. 138-150. DOI: 10.1016/j.ufug.2016.05.008 597
15. Lee, H., Mayer, H., Chen, L. ‘Contribution of trees and grasslands to the mitigation of human heat stress in a residential district 598
of Freiburg, Southwest Germany’. In Landscape and Urban Planning, 2016, Vol. 148, pp. 37-50. DOI: 10.1016/j.landur- 599
bplan.2015.12.004. 600
Appl. Sci. 2021, 11, x FOR PEER REVIEW 22 of 22

16. El-Bardisy, W. M., Fahmy, M., Gohary, G. F. ‘Climatic Sensitive Landscape Design: Towards a Better Microclimate through 601
Plantation in Public Schools, Cairo, Egypt’. Urban Planning and Architecture Design for Sustainable Development, UPADSD 602
14- 16 October 2015. DOI: 10.1016/j.sbspro.2015.12.029. 603
17. Morakinyo, T. E., Lam, Y. F. ‘Simulation study on the impact of tree-configuration, planting pattern and wind condition on 604
street-canyon's micro-climate and thermal comfort’. In Building and Environment, 2016, Vol. 103, pp. 262-275. DOI: 605
10.1016/j.buildenv.2016.04.025. 606
18. Müller, N., Kuttler, W., Barlag, A. B. ‘Counteracting urban climate change: adaptation measures and their effect on thermal 607
comfort’. In Theoretical and Applied Climatology, 2014, Vol. 115, pp. 243–257. DOI: 10.1007/s00704-013-0890-4. 608
19. Vidmar, J., Roset, J. ‘Evaluation of simulation tools for assessment of urban form based on physical performance’. Faculty of 609
Architecture, Ljubljana, Slovenia, 2013. 610
20. Albdour, M.S., Baranyai, B. ‘An overview of microclimate tools for predicting the thermal comfort, meteorological parameters 611
and design strategies in outdoor spaces’. In International Journal for Engineering and Information Sciences, 2019, Vol. 14, No. 2, pp. 612
109–118. DOI: 10.1556/606.2019.14.2.10 613
21. Diéguez, A. P., Duckart, C., Coccolo, S. (2017). ‘Urban thermal comfort study’. Available online: 614
https://whitearkitekter.com/wp-content/uploads/2018/07/Thermal-Comfort-Analysis-Kiruna-White-Arkitekter.pdf (accessed 615
on 07 November 2021). 616
22. Fabbria, K., Nunzioa, A. D., Gasparia, J., Antoninia, E., Boeria, A. ‘Outdoor Comfort: the ENVI-BUG tool to evaluate PMV 617
values Output Comfort point by point’. 8th International Conference on Sustainability in Energy and Buildings, SEB-16, 11-13 618
September 2016, Turin, Italy. DOI: 10.1016/j.egypro.2017.03.213. 619
23. Ibrahim, Y., Kershaw, T., Shepherd, P. ‘A methodology for Modelling Microclimates: A Ladybug-tools and ENVI-met verifica- 620
tion study’. 35th PLEA Conference. Planning Post Carbon Cities, Coruña, Spain, 2020. 621
24. Naboni, E., Meloni, M., Coccolo, S., Kaempf, J., Scartezzini, J. L. ‘An overview of simulation tools for predicting the mean 622
radiant temperature in an outdoor space’. CISBAT 2017 International Conference – Future Buildings & Districts – Energy Effi- 623
ciency from Nano to Urban Scale, CISBAT 2017 6-8 September 2017, Lausanne, Switzerland. DOI: 10.1016/j.egypro.2017.07.471. 624
25. McEvoy, S. (2019). ‘Planning Support Tools in Urban Adaptation Practice’. Doctoral thesis, Delft University of Technology, 625
Netherlands, 2019. DOI: 10.4233/uuid:48b7649c-5062-4c97-bba7-970fc92d7bbf. 626
26. DGNB (2011). ‘DGNB Neubau Stadtquartiere (NSQ) – Kriterien‘. Available online: https://static.dgnb.de/fileadmin/dgnb- 627
ev/de/_archiv/Aktuell/presseinfos/2011/111004_1A_Ausgezeichnete_Stadtquartiere_Kriterien.pdf (accessed on 07 November 628
2021). 629