Towards Flexible Automated Support to Improve the
Quality of Computational Science and Engineering
Software
Davide Falessi, Forrest Shull
Fraunhofer USA
Center for Experimental Software Engineering
College Park, MD, USA
{dfalessi, fshull}@fc-md.umd.edu
Abstract—Continual evolution of the available hardware (e.g. in
terms of increasing size, architecture, and computing power) and
software (e.g. reusable libraries) is the norm rather than exception.
Our goal is to enable CSE developers to spend more of their time
finding scientific results by capitalizing on these evolutions instead of
being stuck in fixing software engineering (SE) problems such as
porting the application to new hardware, debugging, reusing
(unreliable) code, and integrating open source libraries. In this paper
we sketch a flexible automated solution supporting scientists and
engineers in developing accurate and reliable CSE applications. This
solution, by collecting and analyzing product and process metrics,
enables the application of well-established software engineering best
practices (e.g., separation of concerns, regression testing and
inspections) and it is based upon the principles of automation,
flexibility and iteration.
Index Terms—Computational science and engineering software,
empirical software engineering.
I. INTRODUCTION
Events like storms, earthquakes, power plants explosions, and
cancer threaten the lives of billions of people every year.
Computational science and engineering (CSE) software is
clearly of pivotal importance to prevent and forecast these
tragic events and hence minimize human casualties. Continual
evolution of the available hardware (e.g. in terms of increasing
size, architecture, and computing power) and software (e.g.
reusable libraries) is the norm rather than exception. Therefore,
our goal is to enable CSE developers to spend more of their
time finding scientific results by capitalizing on these
evolutions, and less time on fixing software engineering (SE)
problems such as porting the application to new hardware,
debugging, reusing (potentially unreliable) code, and
integrating open source libraries.
Prior work in the CSE domain has shown need for more use
of SE best practices (BPs) but too often heavy practices have
been a poor fit to the domain [1]. Our recent work with
software development teams in the IT field has demonstrated
advances in metrics, automation and iterative approaches that
can effectively semi-automate the detection of instances where
best practices have broken down and corrective action should
be taken [2][3]. The approach, by collecting and analyzing
978-1-4673-6261-0/13/$31.00
c 2013 IEEE
product and process metrics, facilitates the application of well-
established software engineering best practices and is based
upon the principles of automation, flexibility and iteration.
The remainder of this paper describes how this semi-
automated approach can be tailored to the CSE domain:
Section II reports some of the main challenges characterizing
the development of CSE applications. Section III sketches a
flexible automated support to improve the quality of CSE
applications. Section IV concludes the paper.
II. CHALLENGES
Current difficulties in developing accurate and reliable CSE
software include:
• Best Practices (BPs) that are not adequately tailored:
Past research in SE has provided many best practices for
developing robust applications. However, important
differences between CSE applications and more
“traditional” software applications make questionable the
direct application of existing BP to CSE. Specifically, CSE
applications feature complex numerical computations, and
the aim of these applications is to achieve research goals
(e.g., scientific discovery) rather than business goals (e.g.,
increasing revenue). Thus, the direct application of
existing best practices is questionable.
• Difficult V&V. Enacting V&V activities on CSE software
is very difficult because the phenomena under
investigation might be unknown and hence no oracle can
be available. Even when oracles are available, it can be
hard to understand when an error is due to a defect in the
algorithm or in its implementation.
• Education. Many developers of CSE applications have
earned their (PhD) degree in specialized fields, which are
important for applying domain expertise but does not
cover formal software engineering principles and
practices. It is important to build a SE body of knowledge
for CSE applications and reduce practical barriers to their
use.
• Passion about scientific results strides with SE rigor.
CSE researchers clearly strive to reach scientific results in
the shortest time. There is obviously a tradeoff between the
88 SE-CSE 2013, San Francisco, CA, USA
 time spent in developing CSE application of high quality
and the time required to achieve results.
III. TOWARDS A FLEXIBLE SOLUTION
Given the abovementioned challenges, we envision a solution
based upon the following principles:
• Automation: Automation in metrics collection, storage,
and data-mining allows us to easily formalize and transfer
SE knowledge to developers [4].
• Flexibility: In general there is no silver bullet in SE [5].
Thus, we envision a flexible approach that suggests to
developers possible improvements in the application under
development. The “suggestion” mechanism aims to avoid
the strict enforcement of any rules which would make the
developers reject the tool in practice [6].
• Iteration: Our approach requires no big upfront investment
from the developer side and hence facilitates transitioning
towards the application of well-established BPs and the
improvement of our rules. At the end of each iteration, past
suggestions will be used in meetings as the baseline to
collaborate between the metrics team and developers for
adjusting the encoded BP to provide more accurate and
customized suggestions [7].
We propose that similar approaches can work in the CSE
domain as well. We are advocating not to transfer specific BPs
but rather to transfer an approach for adopting, testing and
tweaking BPs.
Figure 1 below reports an overview of a flexible automated
support to improve the quality of CSE applications. Starting
from the top left of Figure 1, we plan to observe the
development activity and collect measures. These measures
will be analyzed by using dynamic thresholds related to
specific rules. Software engineering best practices (e.g.
separation of concern and regression testing) play a vital role in
the approach and are reported in the bottom center of Figure 1.
Specific rules are derived from software engineering best
Data Mining
Figure 1 Overview of a flexible automated support to improve the quality of computational science and engineering software.
89
practices. Further examples of rules and best practices will be
provided in Section IV. Only when all the rules are satisfied
we can deem the related best practice sufficiently applied.
Otherwise (i.e., if the rule is broken), the tool will warn the user
(see the traffic light at the top right of Figure 1).
One of the main novelties of our proposed approach is to
use dynamic thresholds instead of pre-defined thresholds.
Specifically, the threshold for each rule (reported in the top left
of Figure 1) is computed via Data Mining technique(s)
(reported in the center of Figure 1). These techniques,
providing as output the value of a threshold for a specific rule,
take as input:
• Project and product characteristics. CSE applications are
very heterogeneous and thus it is important to tailor the BP
according to the specific project and product
characteristics. According to [1] CSE applications can vary
in terms of:
a. Team size. CSE applications can be developed by
a single developer (i.e. “lone researcher”) for a
small research project or by several developers
(i.e. “Community codes”) even distributed in
different groups and/or different geographic
locations. In general, the higher the number of
developers, the stringent the constraints for the
rules [8].
b. Code life. Small applications (e.g. a code from the
intelligence community) are usually discarded
after a small period of time. In these cases, the
application doesn’t have to be very portable and
efficient and hence the development activity
should not be under stringent constraints. Vice
versa, long living applications will probably be
maintained and ported to new hardware, and hence
stringent constraints must be imposed during the
development activity.
 c. Application size. In general, the higher the size of
the application under development, the higher the
difficulty in its development and maintenance.
Moreover, larger applications, because providing
more features, have a higher likelihood to be
reused (at least partially) in the future. Intuitively
the higher the size, the more stringent constraints
imposed during the development activity.
• Knowledge base. The extent to which thresholds should be
more or less stringent has to be defined not only according
to the project and product characteristics but also
according to historical data.
We note that even the best static analysis tools suffer from a
lack of context, regardless from the size and quality of the
knowledge base. Thus, thresholds, even if dynamic, can be
wrong. Therefore, we will warn the user of broken rules (see
the traffic light at the top right of Figure 1) and we will
deliberately let him/her decide whether to follow the tool
suggestion (of changing something in the application under
development) to meet the rules that are currently broken or let
the application as it is. We note that the use of data mining will
capitalize on this human decision by using the new data to
better adjust future thresholds. This approach aims to address
the abovementioned challenges in developing accurate and
reliable CSE software. Specifically, we address the difficulty of
not having BP adequately tailored for the CSE domain by using
dynamic thresholds and a large set of inputs as size of the
project, etc. This, combined with the use of iterations, allows
the BP to be tailored, step by step, according to specific
application context. We addressed the difficulty of applying
V&V and lack of SE education by automatically checking the
quality of the code. By distilling SE best practices into rules we
transfer the SE knowledge in a viable and inexpensive way to
CSE developers. Finally, the combined use of automation and
flexible output should facilitate the application of SE BPs
without impacting the passion for scientific results. As a matter
of fact, the flexibility allows the CSE developers to decide
when to apply the rule or when the time to achieve scientific
results is more important. Moreover, the automation makes the
BP fast and easy to apply and hence enhance the quality of the
CSE application.
IV. AREAS OF TECHNICAL FOCUS
There are three types of quality that we target:
• Correctness. Less time spent on debugging and reworking
code leaves more time for scientific advancement [1].
Moreover, when scientists have confidence in their code,
they need less time for analyzing whether anomalous
answers are caused by problems in the science or in the
software. So, correct code can also result in less time spent
“debugging” correct science. Measurable indicators of
correct code include: fewer defects being found, less team
effort spent on rework versus adding new functionality.
• Portability. CSE software often migrates to different
platforms to exploit increased computational power or to
different compilers. The primary measurable indicator of
more portable code is less effort (or calendar time)
90
required between the time code is ported to a new platform
and the point at which the code correctly executes.
• Maintainability. Fragile code requires additional effort to
keep up to date, which detracts from the amount of effort
that can be spent on new science. Indicators of improved
maintainability include fewer changes or change requests;
less effort spent per change to the code; fewer classes
affected in each change.
To address these issues, we will develop a solution by focusing
on two software engineering phases that are pertinent to all
CSE software.
A. Architecture/Design
One of the key features of most large, complex software
packages is the presence of an appropriate software architecture
and design. A key software architecture/design practice that is
applicable in this space is separation of concerns. This practice
states that a software system should be modularized in such a
way that portions of the system that are logically independent
should be physically independent within the code base.
Applying this principle to CSE software will help developers
identify code that has not been properly partitioned or
modularized. For example, in a typical CSE system, it would
make sense to separate the scientific portions of the code from
the portions handling parallel communication. This practice
helps with portability because parallel features could be
updated for a new environment without disturbing the science
code. It helps with maintainability and correctness because the
types of V&V required for special-purpose science code and
parallel communication code will be different. Separation of
concerns is a genuine issue. A case study by PIs Hochstein and
Shull [9] demonstrated the difficulties a team had in
maintaining strict separation between the code handling
parallel processing (which required substantial computer
science domain expertise to code correctly) and the rest of the
code base - even when the system was architected to make this
separation explicit. Figure 1 shows a treemap [10] view of a
scientific team’s code base to illustrate the presence of this
problem. Each rectangle represents a source file and the nesting
shows hierarchical organization. The large rectangles outlined
in orange represent packages that were intended to contain the
parallel processing, which is achieved via MPI calls. The
brightness of the boxes indicates the density of MPI calls in
each class. Note that MPI calls are scattered throughout the
code that is not inside the orange boxes. This visualization
helped demonstrate to the team how their architecture rules had
degraded over time in their shared code base. Rules for the
separation of scientific and parallel concerns will define how
these concerns are represented in the architecture of the
software application. For example, if a designated set of
software modules is responsible for taking care of distributing
data and computation in a parallel environment only these
modules are allowed to use the parallel statements of the used
language
B. Verification and Validation (V&V)
As with any software for which the correctness of the results is
important for supporting critical decisions, V&V is an
important, and often times quite difficult, task in most CSE
projects. This difficulty is exacerbated by many factors
including the complexity of the software and the lack of a test
oracle against which to test results.
Figure 2 Distribution of MPI calls in the FLASH code visualized as a treemap.
As one example, regression testing [11] is a method of
developing test cases that can be executed to ensure that code
modifications do not affect the correct execution of that code.
Such a regression test suite may contain a number of unit tests
that can be run relatively quickly, as well as a set of important
integration tests that require more time to run and may be
executed nightly. Regression testing helps with correctness by
demonstrating that code components are functioning correctly
even if other aspects of the code are undergoing substantial or
frequent changes. It helps with maintainability by making
changes that break the existing correct behavior easily spotted.
It helps with portability by easily identifying where the code
breaks when porting to a new environment. This practice
supports the human developers by unloading the responsibility
for validating correctness of all prior work as new science is
being done. Regression testing practices need to be tailored for
the CSE domain, since CSE codes often face the additional
complexity of needing to run tests across multiple architectures
and compilers. This creates additional problems that need to be
addressed in creating, managing, and maintaining regression
test suites. For example, a test suite may be both effective and
useful on an x86-based Linux machine running on gcc, but
refuse to compile (or return slightly different results) when
used on an IBM PowerPC-based system. Detecting such
discrepancies and maintaining test suites that work correctly
across platforms present additional costs that need to be
managed. Rules for regression testing define which parts of the
code are expected to be tested in the parallel environment, and
how complete this testing effort should be. Existing metrics,
such as line, path, and branch coverage will be used to describe
the quality of the testing effort. Example rules could specify
desired levels of code coverage to be exercised by the test suite,
or specify that updates to the test suite should occur
periodically based on the degree of updates to the code itself.
V. CONCLUSIONS
This paper presented the ideas of a flexible automated support
to improve the quality of computational science and
engineering software.
91
Correctness, portability and maintainability are our
priorities and several related software engineering best
practices can be applied for the development of computational
science and engineering software. However, dealing with the
human aspects characterizing the development of CSE
applications still remain one of the main challenge.
Automation, flexibility and iteration seem to be good
ingredients to enable CSE developers to provide accurate and
reliable CSE software.
REFERENCES
[1] V. R. Basili, J. C. Carver, D. Cruzes, L. M. Hochstein, J. K.
Hollingsworth, F. Shull, and M. V. Zelkowitz, “Understanding
the High-Performance-Computing Community: A Software
Engineer’s Perspective,” IEEE Software, vol. 25, no. 4, pp. 29–
36, Jul. 2008.
[2] N. Zazworka, M. A. Shaw, F. Shull, and C. Seaman,
“Investigating the impact of design debt on software quality,” in
Proceeding of the 2nd working on Managing technical debt -
MTD ’11, 2011, p. 17.
[3] J. Schumacher, N. Zazworka, F. Shull, C. Seaman, and M.
Shaw, “Building empirical support for automated code smell
detection,” in Proceedings of the 2010 ACM-IEEE International
Symposium on Empirical Software Engineering and
Measurement - ESEM ’10, 2010, p. 1.
[4] S. Olbrich, D. S. Cruzes, V. Basili, and N. Zazworka, “The
evolution and impact of code smells: A case study of two open
source systems,” in 2009 3rd International Symposium on
Empirical Software Engineering and Measurement, 2009, pp.
390–400.
[5] F. P. Brooks, “No Silver Bullet Essence and Accidents of
Software Engineering,” Computer, vol. 20, no. 4, pp. 10–19,
Apr. 1987.
[6] N. Zazworka, V. R. Basili, and F. Shull, “Tool supported
detection and judgment of nonconformance in process
execution,” in 2009 3rd International Symposium on Empirical
Software Engineering and Measurement, 2009, pp. 312–323.
[7] P. Kruchten, R. L. Nord, and I. Ozkaya, “Technical Debt: From
Metaphor to Theory and Practice,” IEEE Software, vol. 29, no.
6, pp. 18–21, Nov. 2012.
[8] D. Falessi, G. Cantone, S. A. Sarcia’, G. Calavaro, P. Subiaco,
and C. D’Amore, “Peaceful Coexistence: Agile Developer
Perspectives on Software Architecture,” IEEE Software, vol. 27,
no. 2, pp. 23–25, Mar. 2010.
[9] L. Hochstein, F. Shull, and L. B. Reid, “The role of MPI in
development time: A case study,” in 2008 SC - International
Conference for High Performance Computing, Networking,
Storage and Analysis, 2008, pp. 1–10.
[10] L. Hochstein, V. R. Basili, U. Vishkin, and J. Gilbert, “A pilot
study to compare programming effort for two parallel
programming models,” Journal of Systems and Software, vol.
81, no. 11, pp. 1920–1930, Nov. 2008.
[11] H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, “An
empirical study of the effect of time constraints on the cost-
benefits of regression testing,” in Proceedings of the 16th ACM
SIGSOFT International Symposium on Foundations of software
engineering - SIGSOFT ’08/FSE-16, 2008, p. 71.