Software Engineering for Machine Learning:
A Case Study
Saleema Amershi Andrew Begel
Microsoft Research Microsoft Research Microsoft Research
Redmond, WA USA Redmond, WA USA Redmond, WA USA Redmond, WA USA Zurich, Switzerland
samershi@microsoft.com andrew.begel@microsoft.com cbird@microsoft.com rdeline@microsoft.com gall@ifi.uzh.ch
Ece Kamar Nachiappan Nagappan
Microsoft Research Microsoft Research
Redmond, WA USA Redmond, WA USA
eckamar@microsoft.com nachin@microsoft.com besmira.nushi@microsoft.com
Abstract—Recent advances in machine learning have stim-
ulated widespread interest within the Information Technology
sector on integrating AI capabilities into software and services.
This goal has forced organizations to evolve their development
processes. We report on a study that we conducted on observing
software teams at Microsoft as they develop AI-based applica-
tions. We consider a nine-stage workflow process informed by
prior experiences developing AI applications (e.g., search and
NLP) and data science tools (e.g. application diagnostics and bug
reporting). We found that various Microsoft teams have united
this workflow into preexisting, well-evolved, Agile-like software
engineering processes, providing insights about several essential
engineering challenges that organizations may face in creating
large-scale AI solutions for the marketplace. We collected some
best practices from Microsoft teams to address these challenges.
In addition, we have identified three aspects of the AI domain that
make it fundamentally different from prior software application
domains: 1) discovering, managing, and versioning the data
needed for machine learning applications is much more complex
and difficult than other types of software engineering, 2) model
customization and model reuse require very different skills than
are typically found in software teams, and 3) AI components
are more difficult to handle as distinct modules than traditional
software components — models may be “entangled” in complex
ways and experience non-monotonic error behavior. We believe
that the lessons learned by Microsoft teams will be valuable to
other organizations.
Index Terms—AI, Software engineering, process, data
I. I NTRODUCTION
Personal computing. The Internet. The Web. Mobile com-
puting. Cloud computing. Nary a decade goes by without a
disruptive shift in the dominant application domain of the
software industry. Each shift brings with it new software
engineering goals that spur software organizations to evolve
their development practices in order to address the novel
aspects of the domain.
The latest trend to hit the software industry is around
integrating artificial intelligence (AI) capabilities based on
advances in machine learning. AI broadly includes technolo-
gies for reasoning, problem solving, planning, and learning,
among others. Machine learning refers to statistical modeling
Christian Bird Robert DeLine Harald Gall
Microsoft Research University of Zurich
Besmira Nushi Thomas Zimmermann
Microsoft Research Microsoft Research
Redmond, WA USA Redmond, WA USA
tzimmer@microsoft.com
techniques that have powered recent excitement in the software
and services marketplace. Microsoft product teams have used
machine learning to create application suites such as Bing
Search or the Cortana virtual assistant, as well as platforms
such as Microsoft Translator for real-time translation of text,
voice, and video, Cognitive Services for vision, speech, and
language understanding for building interactive, conversational
agents, and the Azure AI platform to enable customers to build
their own machine learning applications [1]. To create these
software products, Microsoft has leveraged its preexisting
capabilities in AI and developed new areas of expertise across
the company.
In this paper, we describe a study in which we learned how
various Microsoft software teams build software applications
with customer-focused AI features. For that, Microsoft has
integrated existing Agile software engineering processes with
AI-specific workflows informed by prior experiences in devel-
oping early AI and data science applications. In our study, we
asked Microsoft employees about how they worked through
the growing challenges of daily software development specific
to AI, as well as the larger, more essential issues inherent in the
development of large-scale AI infrastructure and applications.
With teams across the company having differing amounts of
work experience in AI, we observed that many issues reported
by newer teams dramatically drop in importance as the teams
mature, while some remain as essential to the practice of large-
scale AI. We have made a first attempt to create a process
maturity metric to help teams identify how far they have come
on their journeys to building AI applications.
As a key finding of our analyses, we discovered three funda-
mental differences to building applications and platforms for
training and fielding machine-learning models than we have
seen in prior application domains. First, machine learning is all
about data. The amount of effort and rigor it takes to discover,
source, manage, and version data is inherently more complex
and different than doing the same with software code. Second,
building for customizability and extensibility of models require
teams to not only have software engineering skills but almost
Fig. 1. The nine stages of the machine learning workflow. Some stages are data-oriented (e.g., collection, cleaning, and labeling) and others are model-oriented
(e.g., model requirements, feature engineering, training, evaluation, deployment, and monitoring). There are many feedback loops in the workflow. The larger
feedback arrows denote that model evaluation and monitoring may loop back to any of the previous stages. The smaller feedback arrow illustrates that model
training may loop back to feature engineering (e.g., in representation learning).
always require deep enough knowledge of machine learning to
build, evaluate, and tune models from scratch. Third, it can be
more difficult to maintain strict module boundaries between
machine learning components than for software engineering
modules. Machine learning models can be “entangled” in
complex ways that cause them to affect one another during
training and tuning, even if the software teams building them
intended for them to remain isolated from one another.
The lessons we identified via studies of a variety of teams
at Microsoft who have adapted their software engineering
processes and practices to integrate machine learning can help
other software organizations embarking on their own paths
towards building AI applications and platforms.
In this paper, we offer the following contributions.
1) A description of how several Microsoft software en-
gineering teams work cast into a nine-stage workflow
for integrating machine learning into application and
platform development.
2) A set of best practices for building applications and
platforms relying on machine learning.
3) A custom machine-learning process maturity model for
assessing the progress of software teams towards excel-
lence in building AI applications.
4) A discussion of three fundamental differences in how
software engineering applies to machine-learning–centric
components vs. previous application domains.
II. BACKGROUND
A. Software Engineering Processes
The changing application domain trends in the software
industry have influenced the evolution of the software pro-
cesses practiced by teams at Microsoft. For at least a decade
and a half, many teams have used feedback-intense Agile
methods to develop their software [2], [3], [4] because they
needed to be responsive at addressing changing customer
needs through faster development cycles. Agile methods have
been helpful at supporting further adaptation, for example,
the most recent shift to re-organize numerous team’s prac-
tices around DevOps [5], which better matched the needs
of building and supporting cloud computing applications and
platforms.1 The change to DevOps occurred fairly quickly
because these teams were able to leverage prior capabilities
1 https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/
in continuous integration and diagnostic-gathering, making it
simpler to implement continuous delivery.
Process changes not only alter the day-to-day development
practices of a team, but also influence the roles that people
play. 15 years ago, many teams at Microsoft relied heavily on
development triads consisting of a program manager (require-
ments gathering and scheduling), a developer (programming),
and a tester (testing) [6]. These teams’ adoption of DevOps
combined the roles of developer and tester and integrated
the roles of IT, operations, and diagnostics into the mainline
software team.
In recent years, teams have increased their abilities to an-
alyze diagnostics-based customer application behavior, prior-
itize bugs, estimate failure rates, and understand performance
regressions through the addition of data scientists [7], [8], who
helped pioneer the integration of statistical and machine learn-
ing workflows into software development processes. Some
software teams employ polymath data scientists, who “do it
all,” but as data science needs to scale up, their roles specialize
into domain experts who deeply understand the business prob-
lems, modelers who develop predictive models, and platform
builders who create the cloud-based infrastructure.
B. ML Workflow
One commonly used machine learning workflow at Mi-
crosoft has been depicted in various forms across industry
and research [1], [9], [10], [11]. It has commonalities with
prior workflows defined in the context of data science and data
mining, such as TDSP [12], KDD [13], and CRISP-DM [14].
Despite the minor differences, these representations have in
common the data-centered essence of the process and the
multiple feedback loops among the different stages. Figure 1
shows a simplified view of the workflow consisting of nine
stages.
In the model requirements stage, designers decide which
features are feasible to implement with machine learning and
which can be useful for a given existing product or for a
new one. Most importantly, in this stage, they also decide
what types of models are most appropriate for the given
problem. During data collection, teams look for and integrate
available datasets (e.g., internal or open source) or collect their
own. Often, they might train a partial model using available
generic datasets (e.g., ImageNet for object detection), and then
use transfer learning together with more specialized data to
train a more specific model (e.g., pedestrian detection). Data
cleaning involves removing inaccurate or noisy records from
the dataset, a common activity to all forms of data science.
Data labeling assigns ground truth labels to each record.
For example, an engineer might have a set of images on hand
which have not yet been labeled with the objects present in
the image. Most of the supervised learning techniques require
labels to be able to induce a model. Other techniques (e.g.,
reinforcement learning) use demonstration data or environment
rewards to adjust their policies. Labels can be provided either
by engineers themselves, domain experts, or by crowd workers
in online crowd-sourcing platforms.
Feature engineering refers to all activities that are performed
to extract and select informative features for machine learning
models. For some models (e.g. convolutional neural networks),
this stage is less explicit and often blended with the next stage,
model training. During model training, the chosen models
(using the selected features) are trained and tuned on the
clean, collected data and their respective labels. Then in
model evaluation, the engineers evaluate the output model
on tested or safeguard datasets using pre-defined metrics.
For critical domains, this stage might also involve extensive
human evaluation. The inference code of the model is then
deployed on the targeted device(s) and continuously monitored
for possible errors during real-world execution.
For simplicity the view in Figure 1 is linear, however,
machine learning workflows are highly non-linear and contain
several feedback loops. For example, if engineers notice that
there is a large distribution shift between the training data
and the data in the real world, they might want to go back
and collect more representative data and rerun the workflow.
Similarly, they may revisit their modeling choices made in the
first stage, if the problem evolves or if better algorithms are
invented. While feedback loops are typical in Agile software
processes, the peculiarity of the machine learning workflow
is related to the amount of experimentation needed to con-
verge to a good model for the problem. Indeed, the day-
to-day work of an engineer doing machine learning involves
frequent iterations over the selected model, hyper-parameters,
and dataset refinement. Similar experimental properties have
been observed in the past in scientific software [15] and
hardware/software co-design [16]. This workflow can become
even more complex if the system is integrative, containing
multiple ML components which interact together in complex
and unexpected ways [17].
C. Software Engineering for Machine Learning
The need for adjusting software engineering practices in
the recent era has been discussed in the context of hidden
technical debt [18] and troubleshooting integrative AI [19],
[20]. This work identifies various aspects of ML system archi-
tecture and requirements which need to be considered during
system design. Some of these aspects include hidden feedback
loops, component entanglement and eroded boundaries, non-
monotonic error propagation, continuous quality states, and
mismatches between the real world and evaluation sets. On a
related line of thought, recent work also discusses the impact
that the use of ML-based software has on risk and safety
concerns of ISO standards [21]. In the last five years, there
have been multiple efforts in industry to automate this process
by building frameworks and environments to support the ML
workflow and its experimental nature [1], [22], [23]. However,
ongoing research and surveys show that engineers still struggle
to operationalize and standardize working processes [9], [24],
[23]. The goal of this work is to uncover detailed insights on
ML-specific best practices used by developers at Microsoft.
We share these insights with the broader community aspiring
that such take-away lessons can be valuable to other companies
and engineers.
D. Process Maturity
Software engineers face a constantly changing set of plat-
forms and technologies that they must learn to build the newest
applications for the software marketplace. Some engineers
learn new methods and techniques in school, and bring them
to the organizations they work for. Other learn new skills on
the job or on the side, as they anticipate their organization’s
need for latent talent [25]. Software teams, composed of
individual engineers with varying amounts of experience in the
skills necessary to professionally build ML components and
their support infrastructure, themselves exhibit varying levels
of proficiency in their abilities depending on their aggregate
experience in the domain.
The software engineering discipline has long considered
software process improvement as one of its vital functions.
Researchers and practitioners in the field have developed sev-
eral well-known metrics to assess it, including the Capability
Maturity Model (CMM) [26] and Six Sigma [27]. CMM rates
the software processes of organizations on five levels, from
initial (ad hoc processes), repeatable, defined, capable (i.e.,
quantitatively measured), and efficient (i.e., deliberate process
improvement). Inspired by CMM, we build a first maturity
model for teams building systems and platforms that integrate
machine learning components.
III. S TUDY
We collected data in two phases: an initial set of interviews
to gather the major topics relevant to our research questions
and a wide-scale survey about the identified topics. Our study
design was approved by Microsoft’s Ethics Advisory Board.
A. Interviews
Because the work practice around building and integrating
machine learning into software and services is still emerging
and is not uniform across all product teams, there is no
systematic way to identify the key stakeholders on the topic
of adoption. We therefore used a snowball sampling strategy,
starting with (1) leaders of teams with mature use of machine
learning (ML) (e.g., Bing), (2) leaders of teams where AI is
a major aspect of the user experience (e.g., Cortana), and (3)
people conducting company-wide internal training in AI and
ML. As we chose informants, we picked a variety of teams
 TABLE I responded, giving us a 13.6% response rate. For each open-
T HE STAKEHOLDERS WE INTERVIEWED FOR THE STUDY. response item, between two and four researchers analyzed the
responses through a card sort. Then, the entire team reviewed
Id Role Product Area Manager?
the card sort results for clarity and consistency.
I1 Applied Scientist Search Yes
I2 Applied Scientist Search Yes
Respondents were fairly well spread across all divisions of
I3 Architect Conversation Yes the company and came from a variety of job roles: Data and
I4 Engineering Manager Vision Yes applied science (42%), Software engineering (32%), Program
I5 General Manager ML Tools Yes
I6 Program Manager ML Tools Yes
management (17%), Research (7%), and other (1%). 21% of
I7 Program Manager Productivity Tools Yes respondents were managers and 79% were individual contrib-
I8 Researcher ML Tools Yes utors, helping us balance out the majority manager perspective
I9 Software Engineer Speech Yes
in our interviews.
I10 Program Manager AI Platform No In the next sections, we discuss our interview and survey
I11 Program Manager Community No
I12 Scientist Ads No results, starting with the range of AI applications developed by
I13 Software Engineer Vision No Microsoft, diving into best practices that Microsoft engineers
I14 Software Engineer Vision No have developed to address some of the essential challenges
in building large-scale AI applications and platforms, show-
1. Part 1 ing how the perception of the importance of the challenges
1.1. Background and demographics: changes as teams gain experience building AI applications, and
1.1.1. years of AI experience finally, describing our proposed AI process maturity model.
1.1.2. primary AI use case*
1.1.3. team effectiveness rating IV. A PPLICATIONS OF AI
1.1.4. source of AI components Many teams across Microsoft have augmented their appli-
1.2. Challenges* cations with machine learning and inference, some in some
1.3. Time spent on each of the nine workflow activities surprising domains. We asked survey respondents for the ways
1.4. Time spent on cross-cutting activities that they used AI on their teams. We card sorted this data
2. Part 2 (repeated for two activities where most time spent) twice, once to capture the application domain in which AI
2.1. Tools used* was being applied, and a second time to look at the (mainly)
2.2. Effectiveness rating ML algorithms used to build that application.
2.3. Maturity ratings We found AI is used in traditional areas such as search, ad-
3. Part 3 vertising, machine translation, predicting customer purchases,
3.1. Dream tools* voice recognition, and image recognition, but also saw it
3.2. Best practices* being used in novel areas, such as identifying customer leads,
3.3. General comments* providing design advice for presentations and word processing
documents, providing unique drawing features, healthcare, and
Fig. 2. The structure of the study’s questionnaire. An asterisk indicates an
open-response item. improving gameplay. In addition, machine learning is being
used heavily in infrastructure projects to manage incident
reporting, identify the most likely causes for bugs, monitor
to get different levels of experience and different parts of fraudulent fiscal activity, and to monitor network streams for
the ecosystem (products with AI components, AI frameworks security breaches.
and platforms, AI created for external companies). In all, we Respondents used a broad spectrum of ML approaches to
interviewed 14 software engineers, largely in senior leadership build their applications, from classification, clustering, dy-
roles. These are shown in Table I. The interviews were namic programming, and statistics, to user behavior modeling,
semi-structured and specialized to each informant’s role. For social networking analysis, and collaborative filtering. Some
example, when interviewing Informant I3, we asked questions areas of the company specialized further, for instance, Search
related to his work overseeing teams building the product’s worked heavily with ranking and relevance algorithms along
architectural components. with query understanding. Many divisions in the company
work on natural language processing, developing tools for
B. Survey entity recognition, sentiment analysis, intent prediction, sum-
Based on the results of the interviews, we designed an marization, machine translation, ontology construction, text
open-ended questionnaire whose focus was on existing work similarity, and connecting answers to questions. Finance and
practice, challenges in that work practice, and best practices Sales have been keen to build risk prediction models and do
(Figure 2). We asked about challenges both directly and forecasting. Internal resourcing organizations make use of de-
indirectly by asking informants to imagine “dream tools” and cision optimization algorithms such as resource optimization,
improvements that would make their work practice better. We planning, pricing, bidding, and process optimization.
sent the questionnaire to 4195 members of internal mailing The takeaway for us was that integration of machine learn-
lists on the topics of AI and ML. 551 software engineers ing components is happening all over the company, not just
on teams historically known for it. Thus, we could tell that we
were not just hearing from one niche corner of the company,
but in fact, we received responses from a broad range of
perspectives spread throughout.
V. B EST P RACTICES WITH M ACHINE L EARNING IN
S OFTWARE E NGINEERING
In this section, we present our respondents’ viewpoints on
some of the essential challenges associated with building large-
scale ML applications and platforms and how they address
them in their products. We categorized the challenges by
card sorting interview and survey free response questions, and
then used our own judgment as software engineering and AI
researchers to highlight those that are essential to the practice
of AI on software teams.
A. End-to-end pipeline support
As machine learning components have become more mature
and integrated into larger software systems, our participants
recognized the importance of integrating ML development
support into the traditional software development infrastruc-
ture. They noted that having a seamless development experi-
ence covering (possibly) all the different stages described in
Figure 1 was important to automation. However, achieving this
level of integration can be challenging because of the differ-
ent characteristics of ML modules compared with traditional
software components. For example, previous work in this
field [18], [19] found that variation in the inherent uncertainty
(and error) of data-driven learning algorithms and complex
component entanglement caused by hidden feedback loops
could impose substantial changes (even in specific stages)
which were previously well understood in software engineer-
ing (e.g., specification, testing, debugging, to name a few).
Nevertheless, due to the experimental and even more iterative
nature of ML development, unifying and automating the day-
to-day workflow of software engineers reduces overhead and
facilitate progress in the field.
Respondents report to leverage internal infrastructure in the
company (e.g. AEther2 ) or they have built pipelines specialized
to their own use cases. It is important to develop a “rock solid,
data pipeline, capable of continuously loading and massaging
data, enabling engineers to try out many permutations of AI
algorithms with different hyper-parameters without hassle.”
The pipelines created by these teams are automated, supporting
training, deployment, and integration of models with the
product they are a part of. In addition, some pipeline engineers
indicated that “rich dashboards” showing the value provided
to users are useful.
Several respondents develop openly available IDEs to en-
able Microsoft’s customers to build and deploy their models
(e.g. Azure ML for Visual Studio Code3 and Azure ML
Studio4 ). According to two of our interviewees, the goal
2 https://www.slideshare.net/MSTechCommunity/
ai-microsoft-how-we-do-it-and-how-you-can-too
3 https://marketplace.visualstudio.com/items?itemName=ms-toolsai.vscode-ai
4 https://azure.microsoft.com/en-us/services/machine-learning-studio/
of these environments is to help engineers discover, gather,
ingest, understand, and transform data, and then train, deploy,
and maintain models. In addition, these teams customize the
environments to make them easier to use by engineers with
varying levels of experience. “Visual tools help beginning data
scientists when getting started, but once they know the ropes
and branch out, such tools may get in their way and they may
need something else.”
B. Data availability, collection, cleaning, and management
Since many machine learning techniques are centered
around learning from large datasets, the success of ML-centric
projects often heavily depends on data availability, quality
and management [28]. Labeling datasets is costly and time-
consuming, so it is important to make them available for
use within the company (subject to compliance constraints).
Our respondents confirm that it is important to “reuse the
data as much as possible to reduce duplicated effort.” In
addition to availability, our respondents focus most heavily
on supporting the following data attributes: “accessibility,
accuracy, authoritativeness, freshness, latency, structuredness,
ontological typing, connectedness, and semantic joinability.”
Automation is a vital cross-cutting concern, enabling teams
to more efficiently aggregate data, extract features, synthesize
labelled examples. The increased efficiency enables teams to
“speed up experimentation and work with live data while they
experiment with new models.”
We found that Microsoft teams have found it necessary to
blend data management tools with their ML frameworks to
avoid the fragmentation of data and model management activ-
ities. A fundamental aspect of data management for machine
learning is the rapid evolution of data sources. Continuous
changes in data may originate either from (i) operations
initiated by engineers themselves, or from (ii) incoming fresh
data (e.g., sensor data, user interactions). Either case requires
rigorous data versioning and sharing techniques, for example,
“Each model is tagged with a provenance tag that explains with
which data it has been trained on and which version of the
model. Each dataset is tagged with information about where
it originated from and which version of the code was used to
extract it (and any related features).” This practice is used for
mapping datasets to deployed models or for facilitating data
sharing and reusability.
C. Education and Training
The integration of machine learning continues to become
more ubiquitous in customer-facing products, for example,
machine learning components are now widely used in produc-
tivity software (e.g., email, word processing) and embedded
devices (i.e., edge computing). Thus, engineers with traditional
software engineering backgrounds need to learn how to work
alongside of the ML specialists. A variety of players within
Microsoft have found it incredibly valuable to scaffold their
engineers’ education in a number of ways. First, the company
hosts a twice-yearly internal conference on machine learning
and data science, with at least one day devoted to introductions
to the basics of technologies, algorithms, and best practices.
In addition, employees give talks about internal tools and
the engineering details behind novel projects and product
features, and researchers present cutting-edge advances they
have seen and contributed to academic conferences. Second,
a number of Microsoft teams host weekly open forums on
machine learning and deep learning, enabling practitioners to
get together and learn more about AI. Finally, mailing lists and
online forums with thousands of participants enable anyone
to ask and answer technical and pragmatic questions about
AI and machine learning, as well as frequently share recent
results from academic conferences.
D. Model Debugging and Interpretability
Debugging activities for components that learn from data
not only focus on programming bugs, but also focus on
inherent issues that arise from model errors and uncertainty.
Understanding when and how models fail to make accurate
predictions is an active research area [29], [30], [31], which is
attracting more attention as ML algorithms and optimization
techniques become more complex. Several survey respondents
and the larger Explainable AI community [32], [33] propose
to use more interpretable models, or to develop visualization
techniques that make black-box models more interpretable. For
larger, multi-model systems, respondents apply modularization
in a conventional, layered, and tiered software architecture to
simplify error analysis and debuggability.
E. Model Evolution, Evaluation, and Deployment
ML-centric software goes through frequent revisions initi-
ated by model changes, parameter tuning, and data updates,
the combination of which has a significant impact on system
performance. A number of teams have found it important
to employ rigorous and agile techniques to evaluate their
experiments. They developed systematic processes by adopting
combo-flighting techniques (i.e., flighting a combination of
changes and updates), including multiple metrics in their ex-
periment score cards, and performing human-driven evaluation
for more sensitive data categories. One respondent’s team
uses “score cards for the evaluation of flights and storing
flight information: How long has it been flighted, metrics for
the flight, etc.” Automating tests is as important in machine
learning as it is in software engineering; teams create carefully
put-together test sets that capture what their models should do.
However, it is important that a human remains in the loop. One
respondent said, “we spot check and have a human look at the
errors to see why this particular category is not doing well,
and then hypothesize to figure out problem source.”
Fast-paced model iterations require more frequent deploy-
ment. To ensure that system deployment goes smoothly, sev-
eral engineers recommend not only to automate the training
and deployment pipeline, but also to integrate model building
with the rest of the software, use common versioning reposi-
tories for both ML and non-ML codebases, and tightly couple
the ML and non-ML development sprints and standups.
F. Compliance
Microsoft issued a set of principles around uses of AI in
the open world. These include fairness, accountability, trans-
parency, and ethics. All teams at Microsoft have been asked to
align their engineering practices and the behaviors of fielded
software and services in accordance with these principles.
Respect for them is a high priority in software engineering
and AI and ML processes and practices. A discussion of these
concerns is beyond the scope of this paper. To learn more
about Microsoft’s commitments to this important topic, please
read about its approach to AI.5
G. Varied Perceptions
We found that as a number of product teams at Microsoft in-
tegrated machine learning components into their applications,
their ability to do so effectively was mediated by the amount of
prior experience with machine learning and data science. Some
teams fielded data scientists and researchers with decades of
experience, while others had to grow quickly, picking up their
own experience and more-experienced team members on the
way. Due to this heterogeneity, we expected that our survey
respondents’ perceptions of the challenges their teams’ faced
in practicing machine learning would vary accordingly.
We grouped the respondents into three buckets (low,
medium, and high), evenly divided by the number of years
of experience respondents personally had with AI. First, we
ranked each of the card sorted categories of respondents’
challenges divided by the AI experience buckets. This list is
presented in Table II, initially sorted by the respondents with
low experience with AI.
Two things are worth noticing. First, across the board,
Data Availability, Collection, Cleaning, and Management, is
ranked as the top challenge by many respondents, no matter
their experience level. We find similarly consistent ranking for
issues around the categories of end-to-end pipeline support
and collaboration and working culture. Second, some of the
challenges rise or fall in importance as the respondents’ ex-
perience with AI differs. For example, education and training
is far more important to those with low experience levels in
AI than those with more experience. In addition, respondents
with low experience rank challenges with integrating AI into
larger systems higher than those with medium or high expe-
rience. This means that as individuals (and their teams) gain
experience building applications and platforms that integrate
ML, their increasing skills help shrink the importance of
some of the challenges they perceive. Note, the converse also
occurs. Challenges around tooling, scale, and model evolution,
evaluation, and deployment are more important for engineers
with a lot of experience with AI. This is very likely because
these more experienced individuals are tasked with the more
essentially difficult engineering tasks on their team; those with
low experience are probably tasked to easier problems until
they build up their experience.
5 https://www.microsoft.com/en-us/ai/our-approach-to-ai
 TABLE II
T HE TOP - RANKED CHALLENGES AND PERSONAL EXPERIENCE WITH AI. R ESPONDENTS WERE GROUPED INTO THREE BUCKETS ( LOW, MEDIUM , HIGH )
BASED ON THE 33 RD AND 67 TH PERCENTILE OF THE NUMBER OF YEARS OF AI EXPERIENCE THEY PERSONALLY HAD (N=308). T HE COLUMN Frequency
SHOWS THE INCREASE / DECREASE OF THE FREQUENCY IN THE MEDIUM AND HIGH BUCKETS COMPARED TO THE LOW BUCKETS . T HE COLUMN Rank
SHOWS THE RANKING OF THE CHALLENGES WITHIN EACH EXPERIENCE BUCKET, WITH 1 BEING THE MOST FREQUENT CHALLENGE .

Frequency Rank
Medium High Experience
Challenge vs. Low vs. Low Trend Low Medium High
Data Availability, Collection, Cleaning, and Management -2% 60% 1 1 1
Education and Training -69% -78% 1 5 9
Hardware Resources -32% 13% 3 8 6
End-to-end pipeline support 65% 41% 4 2 4
Collaboration and working culture 19% 69% 5 6 6
Specification 2% 50% 5 8 8
Integrating AI into larger systems -49% -62% 5 16 13
Education: Guidance and Mentoring -83% -81% 5 21 18
AI Tools 144% 193% 9 3 2
Scale 154% 210% 10 4 3
Model Evolution, Evaluation, and Deployment 137% 276% 15 6 4

We also compared the overall frequency of each kind of
challenge using the same three buckets of AI experience.
Looking again at the top ranked challenge, Data Availability,
Collection, Cleaning, and Management, we notice that it
was reported by low and medium experienced respondents at
similar rates, but represented a lot more of the responses (60%)
given by those with high experience. This also happened for
challenges related to Specifications. However, when looking
at Education and Training, Integrating AI into larger systems,
and Education: Guidance and Mentoring, their frequency drops
significantly from the rate reported by the low experience
bucket than reported by the medium and high buckets. We
interpret this to mean that these challenges were less important
to the medium and high experience respondents than to those
with low experience levels. Thus, this table gives a big picture
of both which problems are perceived as most important within
each experience bucket, and which problems are perceived as
most important across the buckets.
Finally, we conducted a logistic regression analysis to build
a model that could explain the differences in frequency when
controlling for personal AI experience, team AI experience,
overall work experience, the number of concurrent AI projects,
and whether or not a respondent had formal education in
machine learning or data science techniques. We found five
significant coefficients:
• Education and Training was negatively correlated with per-
sonal AI experience with a coefficient of -0.18 (p < 0.02),
meaning that people with less AI experience found this to
be a more important issue.
• Educating Others was positively correlated with personal AI
experience with a coefficient of 0.26 (p < 0.01), meaning
that people with greater AI experience found this to be a
more important issue.
• Tool issues are positively correlated with team AI experience
with a coefficient of 0.13 (p < 0.001), meaning that as the
team gains experience working on AI projects, the degree
to which they rely on others’ and their own tools goes up,
making them think about their impact more often.
• End-to-end pipeline support was positively correlated with
formal education (p < 0.01), implying that only those with
formal education were working on building such a pipeline.
• Specifications were also positively correlated with formal
education (p < 0.03), implying that those with formal
education are the ones who write down the specifications
for their models and engineering systems.
The lesson we learn from these analyses is that the kinds
of issues that engineers perceive as important change as
they grow in their experience with AI. Some concerns are
transitory, related to one’s position within the team and the
accidental complexity of working together. Several others are
more fundamental to the practice of integrating machine learn-
ing into software applications, affecting many engineers, no
matter their experience levels. Since machine learning-based
applications are expected to continue to grow in popularity,
we call for further research to address these important issues.
VI. T OWARDS A M ODEL OF ML P ROCESS M ATURITY
As we saw in Section V-G, we see some variance in the
experience levels of AI in software teams. That variation
affects their perception of the engineering challenges to be
addressed in their day-to-day practices. As software teams
mature and gel, they can become more effective and efficient
in delivering machine learning-based products and platforms.
To capture the maturity of ML more precisely than us-
ing a simple years-of-experience number, we created a ma-
turity model with six dimensions evaluating whether each
workflow stage: (1) has defined goals, (2) is consistently
implemented, (3) documented, (4) automated, (5) measured
and tracked, and (6) continuously improved. The factors are
loosely based on the concepts behind the Capability Maturity
Model (CMM) [26] and Six Sigma [27], which are widely
used in software development to assess and improve maturity
of software projects.
In the survey, we asked respondents to report the maturity
for the two workflow stages that each participant spent the
most time on (measured by number of hours they reported
spending on each activity). Specifically, we asked participants
to rate their agreement with the following statements S1 ..S6
(bold text was in the original survey) using a Likert response
format from Strongly Disagree (1) to Strongly Agree (5):
S1: My team has goals defined for what to accomplish with
this activity.
S2: My team does this activity in a consistent manner.
S3: My team has largely documented the practices related
to this activity.
S4: My team does this activity mostly in an automated way.
S5: My team measures and tracks how effective we are at
completing this activity.
S6: My team continuously improves our practices related to
this activity.
We gathered this data for the stages that respondents were
most familiar with because we found that they often specialize
in various stages of the workflow. This question was intended
to be lightweight so that respondents could answer easily,
while at the same time accounting for the wide variety of
ML techniques applied. Rather than being prescriptive (i.e., do
this to get to the next maturity level), our intention was to be
descriptive (e.g., how much automation is there in a particular
workflow stage? how well is a workflow stage documented?).
More work is needed to define maturity levels similar to CMM.
To analyze the responses, we defined an Activity Maturity
Index (AMI) to combine the individual scores into a single
measure. This index is the average of the agreement with
the six maturity statements S1 ..S6 . As a means of validating
the Maturity Index, we asked participants to rate the Activity
Effectiveness (AE) by answering “How effective do you think
your team’s practices around this activity are on a scale from
1 (poor) to 5 (excellent)?”. The Spearman correlation between
the Maturity Index and the Effectiveness was between 0.4982
and 0.7627 (all statistically significant at p < 0.001) for
all AI activities. This suggests that the Maturity Index is a
valid composite measure that can capture the maturity and
effectiveness of AI activities.
In addition to the Activity Maturity Index and Activity
Effectiveness, we collected an Overall Effectiveness (OE)
score by asking respondents the question “How effectively
does your team work with AI on a scale from 1 (poor) to 5
(excellent)” Having the AMI, AE, and OE measures allowed
us to compare the maturity and effectiveness of different
organizations, disciplines, and application domains within Mi-
crosoft, and identify areas for improvement. We plot one of
these comparisons in Figure 3 and show the average overall
effectiveness scores divided by nine of the most represented
AI application domains in our survey. There are two things
to notice. First, the spread of the y-values indicates that
the OE metric can numerically distinguish between teams,
Fig. 3. The average overall effectiveness (OE) of a team’s ML practices
divided by application domain (anonymized). The y-axis labels have been
elided for confidentiality. An ANOVA and Scott Knott test identified two
distinct groups to the OE metric, labeled in black (A–F) and red (G–I).
meaning that some respondents feel their teams are at different
levels of maturity than others. Second, an ANOVA and Scott
Knott test show significant differences in the reported values,
demonstrating the potential value of this metric to identify the
various ML process maturity levels.
We recognize that these metrics represent a first attempt at
quantifying a process metric to enable teams to assess how
well they practice ML. In future work, we will refine our
instrument and further validate its utility.
VII. D ISCUSSION
In this section, we synthesize our findings into three ob-
servations of some fundamental differences in the way that
software engineering has been adapted to support past popular
application domains and how it can be adapted to support
artificial intelligence applications and platforms. There may
be more differences, but from our data and discussions with
ML experts around Microsoft, these three rose to prominence.
A. Data discovery and management
Just as software engineering is primarily about the code
that forms shipping software, ML is all about the data that
powers learning models. Software engineers prefer to design
and build systems which are elegant, abstract, modular, and
simple. By contrast, the data used in machine learning are vo-
luminous, context-specific, heterogeneous, and often complex
to describe. These differences result in difficult problems when
ML models are integrated into software systems at scale.
Engineers have to find, collect, curate, clean, and process
data for use in model training and tuning. All the data has
to be stored, tracked, and versioned. While software APIs
are described by specifications, datasets rarely have explicit
schema definitions to describe the columns and characterize
their statistical distributions. However, due to the rapid itera-
tion involved in ML, the data schema (and the data) change
frequently, even many times per day. When data is ingested
from large-scale diagnostic data feeds, if ML engineers want
to change which data values are collected, they must wait
for the engineering systems to be updated, deployed, and
propagated before new data can arrive. Even “simple” changes
can have significant impacts on the volume of data collected,
potentially impacting applications through altered performance
characteristics or increased network bandwidth usage.
While there are very well-designed technologies to version
code, the same is not true for data. A given data set may
contain data from several different schema regimes. When a
single engineer gathers and processes this data, they can keep
track of these unwritten details, but when project sizes scale,
maintaining this tribal knowledge can become a burden. To
help codify this information into a machine-readable form,
Gebru et al. propose to use data sheets inspired by elec-
tronics to more transparently and reliably track the metadata
characteristics of these datasets [34]. To compare datasets
against each other, the Datadiff [35] tool enables developers to
formulate viable transformation functions over data samples.
B. Customization and Reuse
While it is well-understood how much work it takes to cus-
tomize and reuse code components, customizing ML models
can require much more. In software, the primary units of reuse
are functions, algorithms, libraries, and modules. A software
engineer can find the source code for a library (e.g. on Github),
fork it, and easily make changes to the code, using the same
skills they use to develop their own software.
Although fully-trained ML models appear to be functions
that one can call for a given input, the reality is far more
complex. One part of a model is the algorithm that powers
the particular machine learning technique being used (e.g.,
SVM or neural nets). Another is the set of parameters that
controls the function (e.g., the SVM support vectors or neural
net weights) and are learned during training. If an engineer
wants to apply the model on a similar domain as the data it was
originally trained on, reusing it is straightforward. However,
more signficant changes are needed when one needs to run
the model on a different domain or use a slightly different
input format. One cannot simply change the parameters with
a text editor. In fact, the model may require retraining, or
worse, may need to be replaced with another model. Both
require the software developer to have machine learning skills,
which they may never have learned. Beyond that, retraining
or rebuilding the model requires additional training data to be
discovered, collected, and cleaned, which can take as much
work and expertise as the original model’s authors put in.
C. ML Modularity
Another key attribute of engineering large-scale software
systems is modularity. Modules are separated and isolated
to ensure that developing one component does not interfere
with the behavior of others under development. In addition,
software modularity is strengthened by Conway’s Law, which
makes the observation that the teams that build each com-
ponent of the software organize themselves similarly to its
architecture. Thus, separate modules are often assigned to
separate teams. Module interactions are controlled by APIs
which do dual duty to enable software modules to remain
apart, but also describe the interfaces to minimize the amount
of communication needed between separate teams to make
their modules work together [36], [37].
Maintaining strict module boundaries between machine
learned models is difficult for two reasons. First, models are
not easily extensible. For example, one cannot (yet) take an
NLP model of English and add a separate NLP model for
ordering pizza and expect them to work properly together.
Similarly, one cannot take that same model for pizza and pair
it with an equivalent NLP model for French and have it work.
The models would have to be developed and trained together.
Second, models interact in non-obvious ways. In large-
scale systems with more than a single model, each model’s
results will affect one another’s training and tuning processes.
In fact, one model’s effectiveness will change as a result of
the other model, even if their code is kept separated. Thus,
even if separate teams built each model, they would have to
collaborate closely in order to properly train or maintain the
full system. This phenomenon (also referred to as component
entanglement) can lead to non-monotonic error propagation,
meaning that improvements in one part of the system might
decrease the overall system quality because the rest of the
system is not tuned to the latest improvements. This issue is
even more evident in cases when machine learning models are
not updated in a compatible way and introduce new, previously
unseen mistakes that break the interaction with other parts of
the system which rely on it.
VIII. L IMITATIONS
Our case study was conducted with teams at Microsoft, a
large, world-wide software company with a diverse portfolio
of software products. It is also one of the largest purveyors of
machine learning-based products and platforms. Some findings
are likely to be specific to the Microsoft teams and team
members who participated in our interviews and surveys. Nev-
ertheless, given the high variety of projects represented by our
informants, we expect that many of the lessons we present in
this paper will apply to other companies. Some of our findings
depend on the particular ML workflow used by some software
teams at Microsoft. The reader should be able to identify how
our model abstractions fit into the particulars of the models
they use. Finally, interviews and surveys rely on self-selected
informants and self-reported data. Wherever appropriate, we
stated that findings were our informants’ perceptions and
opinions. This is especially true with this implementation
of our ML process maturity model, which triangulated its
measures against other equally subjective measures with no
objective baseline. Future implementations of the maturity
model should endeavor to gather objective measures of team
process performance and evolution.
 IX. C ONCLUSION
Many teams at Microsoft have put significant effort into
developing an extensive portfolio of AI applications and plat-
forms by integrating machine learning into existing software
engineering processes and cultivating and growing ML talent.
In this paper, we described the results of a study to learn
more about the process and practice changes undertaken by a
number of Microsoft teams in recent years. From these find-
ings, we synthesized a set of best practices to address issues
fundamental to the large-scale development and deployment of
ML-based applications. Some reported issues were correlated
with the respondents’ experience with AI, while others were
applicable to most respondents building AI applications. We
presented a ML process maturity metric to help teams self-
assess how well they work with machine learning and offer
guidance towards improvements. Finally, we identified three
aspects of the AI domain that make it fundamentally different
than prior application domains. Their impact will require
significant research efforts to address in the future.
R EFERENCES
[1] M. Salvaris, D. Dean, and W. H. Tok, “Microsoft AI Platform,” in Deep
Learning with Azure. Springer, 2018, pp. 79–98.
[2] A. Begel and N. Nagappan, “Usage and perceptions of agile software
development in an industrial context: An exploratory study,” in First
International Symposium on Empirical Software Engineering and Mea-
surement (ESEM 2007), Sept 2007, pp. 255–264.
[3] A. Begel and N. Nagappan, “Pair programming: What’s in it for me?” in
Proc. of the Second ACM-IEEE International Symposium on Empirical
Software Engineering and Measurement, 2008, pp. 120–128.
[4] B. Murphy, C. Bird, T. Zimmermann, L. Williams, N. Nagappan, and
A. Begel, “Have agile techniques been the silver bullet for software
development at microsoft?” in 2013 ACM/IEEE Intl. Symp. on Empirical
Software Engineering and Measurement, Oct 2013, pp. 75–84.
[5] M. Senapathi, J. Buchan, and H. Osman, “DevOps capabilities, practices,
and challenges: Insights from a case study,” in Proc. of the 22nd
International Conference on Evaluation and Assessment in Software
Engineering 2018, 2018, pp. 57–67.
[6] T. D. LaToza, G. Venolia, and R. DeLine, “Maintaining mental models:
A study of developer work habits,” in Proc. of the 28th International
Conference on Software Engineering, 2006, pp. 492–501.
[7] M. Kim, T. Zimmermann, R. DeLine, and A. Begel, “The emerging role
of data scientists on software development teams,” in Proc. of the 38th
International Conference on Software Engineering, 2016, pp. 96–107.
[8] M. Kim, T. Zimmermann, R. DeLine, and A. Begel, “Data scientists in
software teams: State of the art and challenges,” IEEE Transactions on
Software Engineering, vol. 44, no. 11, pp. 1024–1038, 2018.
[9] C. Hill, R. Bellamy, T. Erickson, and M. Burnett, “Trials and tribulations
of developers of intelligent systems: A field study,” in Visual Languages
and Human-Centric Computing (VL/HCC), 2016 IEEE Symposium on.
IEEE, 2016, pp. 162–170.
[10] “Machine learning workflow,” https://cloud.google.com/ml-
engine/docs/tensorflow/ml-solutions-overview, accessed: 2018-09-24.
[11] K. Patel, J. Fogarty, J. A. Landay, and B. Harrison, “Investigating
statistical machine learning as a tool for software development,” in Proc.
of the SIGCHI Conference on Human Factors in Computing Systems.
ACM, 2008, pp. 667–676.
[12] “The Team Data Science Process,” https://docs.microsoft.com/en-
us/azure/machine-learning/team-data-science-process/, accessed: 2018-
09-24.
[13] U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth, “The KDD process for
extracting useful knowledge from volumes of data,” Communications of
the ACM, vol. 39, no. 11, pp. 27–34, 1996.
[14] R. Wirth and J. Hipp, “CRISP-DM: Towards a standard process model
for data mining,” in Proc. 4th Intl. Conference on Practical Applications
of Knowledge Discovery and Data mining, 2000, pp. 29–39.
[15] J. E. Hannay, C. MacLeod, J. Singer, H. P. Langtangen, D. Pfahl, and
G. Wilson, “How do scientists develop and use scientific software?”
in Proc. of the 2009 ICSE workshop on Software Engineering for
Computational Science and Engineering. IEEE Computer Society,
2009, pp. 1–8.
[16] G. De Michell and R. K. Gupta, “Hardware/software co-design,” Proc.
of the IEEE, vol. 85, no. 3, pp. 349–365, 1997.
[17] D. Bohus, S. Andrist, and M. Jalobeanu, “Rapid development of
multimodal interactive systems: a demonstration of platform for situated
intelligence,” in Proc. of the 19th ACM International Conference on
Multimodal Interaction. ACM, 2017, pp. 493–494.
[18] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner,
V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden
technical debt in machine learning systems,” in NIPS, 2015.
[19] B. Nushi, E. Kamar, E. Horvitz, and D. Kossmann, “On human intellect
and machine failures: Troubleshooting integrative machine learning
systems.” in AAAI, 2017, pp. 1017–1025.
[20] S. Andrist, D. Bohus, E. Kamar, and E. Horvitz, “What went wrong
and why? diagnosing situated interaction failures in the wild,” in ICSR.
Springer, 2017, pp. 293–303.
[21] R. Salay, R. Queiroz, and K. Czarnecki, “An analysis of ISO 26262:
Using machine learning safely in automotive software,” arXiv preprint
arXiv:1709.02435, 2017.
[22] D. Baylor, E. Breck, H.-T. Cheng, N. Fiedel, C. Y. Foo, Z. Haque,
S. Haykal, M. Ispir, V. Jain, L. Koc et al., “TFX: A tensorflow-based
production-scale machine learning platform,” in Proc. of the 23rd ACM
SIGKDD. ACM, 2017, pp. 1387–1395.
[23] V. Sridhar, S. Subramanian, D. Arteaga, S. Sundararaman, D. Roselli,
and N. Talagala, “Model governance: Reducing the anarchy of produc-
tion ML,” in USENIX. USENIX Association, 2018, pp. 351–358.
[24] T. Wuest, D. Weimer, C. Irgens, and K.-D. Thoben, “Machine learning
in manufacturing: advantages, challenges, and applications,” Production
& Manufacturing Research, vol. 4, no. 1, pp. 23–45, 2016.
[25] J. Sillito and A. Begel, “App-directed learning: An exploratory study,”
in 6th International Workshop on Cooperative and Human Aspects of
Software Engineering (CHASE), May 2013, pp. 81–84.
[26] C. Weber, B. Curtis, and M. B. Chrissis, The capability maturity model,
guidelines for improving the software process. Harlow: Addison Wesley,
1994.
[27] M. Alexander, Six Sigma: The breakthrough management strategy rev-
olutionizing the world’s top corporations. Taylor & Francis, 2001.
[28] N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, “Data manage-
ment challenges in production machine learning,” in Proc. of the 2017
ACM SIGMOD, 2017, pp. 1723–1726.
[29] T. Kulesza, M. Burnett, W.-K. Wong, and S. Stumpf, “Principles of
explanatory debugging to personalize interactive machine learning,” in
Proc. of the 20th International Conference on Intelligent User Interfaces,
2015, pp. 126–137.
[30] S. Amershi, M. Chickering, S. M. Drucker, B. Lee, P. Simard, and J. Suh,
“Modeltracker: Redesigning performance analysis tools for machine
learning,” in Proc. of the 33rd Annual ACM Conference on Human
Factors in Computing Systems, 2015, pp. 337–346.
[31] B. Nushi, E. Kamar, and E. Horvitz, “Towards accountable AI: Hybrid
human-machine analyses for characterizing system failure,” in HCOMP,
2018, pp. 126–135.
[32] D. Gunning, “Explainable artificial intelligence (XAI),” Defense Ad-
vanced Research Projects Agency (DARPA), 2017.
[33] D. S. Weld and G. Bansal, “Intelligible artificial intelligence,” arXiv
preprint arXiv:1803.04263, 2018.
[34] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. M. Wallach,
H. D. III, and K. Crawford, “Datasheets for datasets,” CoRR, vol.
abs/1803.09010, 2018.
[35] C. Sutton, T. Hobson, J. Geddes, and R. Caruana, “Data diff: Inter-
pretable, executable summaries of changes in distributions for data
wrangling,” in Proc. of the 24th ACM SIGKDD. ACM, 2018, pp.
2279–2288.
[36] C. R. B. de Souza, D. Redmiles, and P. Dourish, ““Breaking the Code”,
moving between private and public work in collaborative software
development,” in Proc. of the 2003 International ACM SIGGROUP
Conference on Supporting Group Work, 2003, pp. 105–114.
[37] C. R. B. de Souza, D. Redmiles, L.-T. Cheng, D. Millen, and J. Patterson,
“Sometimes you need to see through walls: A field study of application
programming interfaces,” in Proc. of the 2004 ACM Conference on
Computer Supported Cooperative Work, 2004, pp. 63–71.
 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops

A Survey of Software Quality for Machine

Learning Applications
Satoshi Masuda, Kohichi Ono, Toshiaki Yasue, Nobuhiro Hosokawa
IBM Research - Tokyo
Chuou-ku, Tokyo, Japan 103-8510
Email: {smasuda, onono, yasue, carvin}@jp.ibm.com

Abstract—Machine learning (ML) is now widespread. Tradi- A. Survey objective

tional software engineering can be applied to the development
ML applications. However, we have to consider specific problems
with ML applications in therms of their quality. In this paper,
we present a survey of software quality for ML applications
to consider the quality of ML applications as an emerging
discussion. From this survey, we raised problems with ML
applications and discovered software engineering approaches and
software testing research areas to solve these problems. We
classified survey targets into Academic Conferences, Magazines,
and Communities. We targeted 16 academic conferences on
artificial intelligence and software engineering, including 78
papers. We targeted 5 Magazines, including 22 papers. The
results indicated key areas, such as deep learning, fault localization,
and prediction, to be researched with software engineering and
testing.
Index Terms—Machine Learning, Software Quality, Software
Engineering and Testing
I. I NTRODUCTION
Machine learning (ML) applications, such as face recog-
nition, question answering, and sales analysis, are now
widespread. Many ML services are available from APIs on
the cloud and are called ML-as-a-services (MLaaS). ML
applications often consist of such MLaaS. When we develop
ML applications based on requirements, traditional software
engineering can be applied. However, we have to consider
specific problems with ML applications in terms of their
The main objectives of the survey is to discover techniques
to evaluate and improve the software quality of ML applica-
tions, and discuss future software testing research area. The
sub objective of the survey is to find solutions to software
testing problems for ML applications. Figure 1 shows an
example of an ML application and problems in software
quality for that ML applications from industry experiences.
We focus the following problems:
• How to verify the answer that MLaaS return to unknown
data.
• How to verify insufficient or biased data because prob-
lems because data determine the logic of ML applica-
tions.
• How to verify the quality of end-to-end systems.
• How to notify users confidence of answer correctness
from the system.
These problems have not been solved by traditional software
engineering or testing, and are candidates of research area for
research in software testing.
B. Survey targets
The 16 targeted academic conferences on artificial intelli-
gence and software engineering are as follows:
• AAAI: Association for the Advancement of Artificial

quality. The problems are that training data determines the Intelligence
logic of ML applications, and the results of ML application • ASE: IEEE/ACM International Conference on Automated

to unknown data cannot be verified in terms of correctness. Software Engineering

Hence, new software engineering approaches are required to • CVPR: IEEE Conference on Computer Vision and Pattern

solve the problems. In this paper, we present a survey of Recognition

software quality for ML applications to consider the quality
of ML applications. From the survey, we raised problems Public cloud Private cloud Enterprise network
with ML applications and discovered software engineering How to verify quality of end to
approaches and software testing research areas to solve these How to notify
confidence of
Data Data end system

problems. Userss
answer?
MLaaS
M MLaaS insufficient Enterprise
E t i
or biased Application
We classified survey targets into Academic Conferences, How to verify data
ML answer to
Magazines, and Communities. We targeted 16 academic con- unknown data

ferences on artificial intelligence and software engineering, Mobile Cloud Transaction Enterprise
Application Messaging Connectivity Process
including 78 papers. We targeted 5 Magazines, including 22
papers.
: Actors of application : Messaging : Software testing research question
II. P ROCEDURE OF SURVEY
We present the procedure of our survey in this section. Fig. 1. ML Application and software testing research questions

0-7695-6432-1/18/$31.00 ©2018 IEEE 279

DOI 10.1109/ICSTW.2018.00061
 • FAS*W: Foundations and Applications of Self* Systems,
IEEE International Workshops on
• ICML: International Conference on Machine Learning
• ICST: IEEE International Conference on Software Test-
ing, Verification and Validation
• ICSTW: Workshop on IEEE International Conference on
Software Testing, Verification and Validation
• IEEE Transaction: IEEE Transactions on Software Engi-
neering
• IJCAI: International Joint Conference on Artificial Intel-
ligence
• ISSTA: ACM SIGSOFT International Symposium on
Software Testing and Analysis
• ITAW: Information Theory and Applications Workshop
• KDD: ACM SIGKDD Conference on Knowledge Dis-
covery and Data Mining
• MELECON: Mediterranean Electrotechnical Conference
• NIPS: Annual Conference on Neural Information Pro-
cessing Systems
• PUC: Personal and Ubiquitous Computing
• SOSP: ACM Symposium on Operating Systems Princi-
ples
The five targeted magazines are as follows:
• AI Magazine
• IEEE Spectrum
• MIT news - Artificial intelligence
• Analytics Magazine
• OpenAI
The two targeted communities are the developer communi-
ties of software engineers.
C. Survey Steps
We executed the survey in the following steps:
1) Search articles by retrieving key words: quality, verifi-
cation, engineering, design and other words related to
software engineering.
2) Read articles.
3) Tag one key word to articles to classify them. The tags
are based on the Machine Learning Dictionary [1] and
Artificial Intelligence Vocabulary [2]. If we could not
find an appropriate word in the vocabulary, we created
a tag from the contents of the article. Some academic
papers in the articles have key words in their contents;
however, the key words indicate so extensive knowledge
that we could not classify the articles.
4) Make comments and evaluate. We evaluated the articles
as representative papers R or subsidiary papers S.
III. S URVEY R ESULTS
We present the survey results in this section and summarize
the methods and techniques that described in the articles.
A. Results
Table I lists the conferences and the number of articles.
The total number of articles was 101 from the search results.
This number was too small with respect to articles in each
280
conferences. For instance, about 700 papers were accepted
at AAAI2017. This small number indicates that the topics
of quality, verification, engineering, and design are not yet
major topics. Table II lists tags and the number of tags for
each evaluations sorted by the number of evaluations. Deep
learning was the top-ranked tag. Deep learning has been the
most important technique in ML, so there were many articles
related to the search words quality, verification, engineering
and design. The second-ranked tag was Fault localization,
and the third was Prediction. Fault localization is a software
engineering term, so it indicates that software engineering may
shift to ML.
B. Software Quality methods and techniques for ML applica-
tions
We summarize the software-quality methods and techniques
for ML applications from the top 7 ranked tags in the survey
results.
1) Deep Learning: Pei et al. discussed how to find a
”corner case” (case rarely occurs, a troublesome case) in a
deep learning system and a test method using it [80]. Hsu
mentioned the method in the article ”A New Way to Find Bugs
in Self-Driving AI Could Save Lives” [96]. Bau et al. proposed
a technique for interpreting what internal firing of a deep
learning model corresponds to using semantic segmentation
[28]. Schulam et al. proposed a method of generating counter-
factual predictions [73]. The method targeted a circumstances
that if prediction can be executed by using experience-based
data the predictions have good results, but experience-based
data cannot be always used due to ethical problems. Zhang
et al., Sangkloy et al. proposed verification method of visual
pattern in deep learning [14], [19]. Kokkinos et al. discussed an
innovative method of deep learning. They proposed a method
that train in an end-to-end manner a convolutional neural
network (CNN) that jointly handles low-, mid-, and high-level
vision tasks in a unified architecture [22].
2) Fault Localization: Wong et al. surveyed software fault
localization [44]. Software fault localization, the act of iden-
tifying the locations of faults in a program, is widely rec-
ognized as one of the most tedious, time consuming, and
expensive activities in program debugging. Fault localization
is challenging for ML applications [59]. Le et al. proposed
a new fault localization approach that employs a learning-to-
rank strategy. The approach includes ranking methods based
on their likelihood of being a root cause of a failure. Sun et al.
investigated several coverage-based statistical fault localization
metrics and reported on an empirical study they conducted to
assess the relative importance of those metrics elements [37].
3) Prediction: Prediction and inference are important func-
tions of applications with ML applications. As software sys-
tems increase in complexity and operate with less human
supervision, it becomes more difficult to use traditional tech-
niques to detect when software is not behaving as intended
[62]. For instance, predicting human behavior in front of
software systems helps better design software. Katz et al. pro-
posed techniques to model the behavior of executing programs
 TABLE I TABLE II
N UMBER OF ARTICLES N UMBER OF TAGS

Category Name Number References Tags R 1 S 2 Sub Total

of Deep learning 3 6 9
articles Fault localization 3 3
Academic AAAI2016 3 [3], [4], [5] Prediction 3 3
conferences AAAI2017 4 [6], [7], [8], [9] MLaaS 2 3 5
ASE2015 1 [10] Multi agent 2 2 4
ASE2016 3 [11], [12], [13] Search-Based 2 2 4
CVPR017 17 [14], [15], [16], [17], [18], Model checking 2 1 3
[19], [20], [21], [22], [23], AI business 1 7 8
[24], [25], [26], [27], [28], Genetic algorithm 1 1 2
[29], [30] Robot 1 1 2
FAS*W2016 1 [31] Technical debt 1 1
ICML2015 1 [32] Democratizing AI 1 1
ICML2017 1 [33] AI ethics 1 1
ICST2015 3 [34], [35], [36] Continuous integration 1 1
ICST2016 3 [37], [38], [39] Adversarial 1 1
ICST2017 3 [40], [41], [42] AI for complex 1 1
ICSTW2011 1 [43] Lifelong Learning Machines 1 1
IEEE Trans- 1 [44] Graph quality 1 1
actions Testing 7 7
IJCAI2016 4 [45], [46], [47], [48] Neural network 7 7
IJCAI2017 6 [49], [50], [51], [52], [53], Safety 7 7
[54] Software development 6 6
ISSTA2013 3 [55], [56], [57] Software engineering 4 4
ISSTA2016 2 [58], [59] AI overview 3 3
ISSTA2017 4 [60], [61], [62], [63] Testing automation 2 2
ITAW2017 1 [64] Symbolic execution 1 1
KDD2016 2 [65], [66] Static analysis 1 1
KDD2017 2 [67], [68] Timed failure 1 1
MELECON16 1 [69] Auto encoder 1 1
NIPS2015 1 [70] Pairwise 1 1
NIPS2017 8 [71], [72], [73], [74], [75], Algorithm design 1 1
[76], [77], [78] Security 1 1
PUC2016 1 [79] Training data 1 1
SOSP2017 1 [80] Speaker verification 1 1
Community Commercial 1 [81] Linear dynamic logic 1 1
Developer 1 [82] 3d model 1 1
Magazines AI Magazine 9 [83], [84], [85], [86], [87], Mechanism design 1 1
[88], [89], [90], [91] Label data 1 1
Business In- 1 [92] Metrics 1 1
sider Total 28 73 101
IEEE Spec- 5 [93], [94], [95], [96], [97]
tram
1. R: Representative papers, 2. S: Subsidiary papers
Open AI 6 [98], [99], [100], [101],
[102], [103]
Total 101
application, Bosse et al. developed distributed ML with self-
organizing mobile agents for earthquake monitoring [31].
using low-level signals collected during executions [62]. The 5) Multi Agent: Multi agent is a key approach to adapt soft-
models provide a basis for predicting whether an execution ware engineering into ML applications such as autonomous
of a program or program unit under test represents intended vehicles and robots. Verifying a data-aware multi-agent system
behavior. Hotzkow et al. presented automatically inferring (DAMAS) is challenging because of the infinite state models
user expectations from the semantic contexts over multiple generated by their infinite-domain variables. Belardinelli et al.
applications [63]. Once the user expectations are established, proposed a method of parameterized DAMAS (P-DAMAS)
this knowledge can be used as an oracle as a system with an unbounded number of homogenous
4) MLaaS: Ribeiro et al. defined MLaaS as a ready-to-use agents, each assumed to be data-aware, i.e., endowed with
ML functions. They proposed an architecture to create a flexi- possibly infinite domains and interacting with an environment
ble and scalable MLaaS [32]. VanDerHerten et al. developed a composed of partially shared data [53]. Wooldridge et al.
proof of concept for adaptive modeling and sampling method- presented formal models through which rational verification
ologies for Internet-of-Things applications with MLaaS [69]. can be studied, and surveyed the complexity of key decision
Assem et al. developed a methodology to build an application problems in multi-agent systems [4].
with train, compare, decide, and change approach with MLaaS 6) Search-Based: Search-Based Software Engineering is
[79]. This is more practical than theoretical; however, it seems the name given to a body of work in which search-based
to be the most helpful reference in the real world. For concrete optimization is applied to software engineering [34]. McMinn

281

C ORRESPONDENCE OF PROBLEMS AND TAGS
Problems
How to verify the answer that MLaaS return to unknown data
How to verify insufficient or biased data because problems be-
cause data determine the logic of ML applications.
How to verify the quality of end-to-end systems.
How to notify users confidence of answer correctness from the
system.
et al. presented Search-Based Software Testing, which is the
use of a meta-heuristic optimizing search technique, such as a
Genetic Algorithm, to automate or partially automate a testing
task; for example, the automatic generation of test data [43].
Gay2017 proposed assessed search-based generation of test
suites that detect real faults [42].
7) Model Checking: Model checking is a traditional tech-
nique of software engineering. However, describing applica-
tions models with ML allows the application model checking
to verify the applications. Alechina et al. proposed a method
for verifying existence of resource-bounded coalition uniform
strategies to be able to automatically verify properties of such
systems using model-checking [47]. Tappler et al. presented
learning-based approach to detecting failures by model-based
testing [40].
IV. D ISCUSSION
Table III shows articles that have correspondence between
problems and top seven tags. For instance, Pei et al. [80]
discussed a corner-case in a deep learning system and a white-
box testing method using the corner-case. The corner-case
is an unknown data for ML applications; hence, their paper
corresponds to the problem. We put corresponding papers to
each problems in the same manner. As summarized in Section
3, we found several techniques or methods of software quality
for ML applications. Some of these techniques corresponded to
problems that we raised. However, the problems have not been
completely solved. Each papers covered some of the problems.
For instance, Pei et al. targeted deep learning systems but
did not target other ML models such as k-means, decision-
tree, and support vector machine. ML applications consist of
various ML models; therefore, techniques of software quality
or testing are required for verifying of ML applications.
V. C ONCLUSION
ML applications such as face recognition, question answer-
ing, and sales analysis are now widespread. New software
engineering approaches are required to solve the problems.
We presented a survey of software quality for ML applications
to consider the quality of ML applications. From our survey
determined problems with ML applications and discovered
software engineering approaches and software testing research
areas to solve these problems.
TABLE III
Tags
Deep Fault Prediction MLaaS Multi Search- Model
learning localiza- agent Based check-
tion ing
[80] [62] [40]
[73] [53]
[59] [79] [4] [42] [47]
[63],
[66]
R EFERENCES
[1] B. Wilson, “The Machine Learning Dictionary,” 2012. [Online].
Available: http://www.cse.unsw.edu.au/ billw/mldict.html
[2] G. S. Novak Jr., “Artificial Intelligence Vocabulary,” 2005. [Online].
Available: https://www.cs.utexas.edu/users/novak/aivocab.html
[3] B. Zarrieß and J. Claßen, “Decidable Verification of Golog Programs
over Non-Local Effect Actions,” pp. 1109–1115.
[4] M. Wooldridge, J. Gutierrez, P. Harrenstein, E. Marchioni, G. Perelli,
and A. Toumi, “Rational Verification: From Model Checking to Equi-
librium Checking,” Proceedings of the 30th Conference on Artificial
Intelligence (AAAI 2016), pp. 4184–4190, 2016.
[5] B. Bittner, M. Bozzano, A. Cimatti, and G. Zampedri, “Automated
Verification and Tightening of Failure Propagation Models,” Proceed-
ings of the 30th Conference on Artificial Intelligence (AAAI 2016), pp.
907–913, 2016.
[6] M. Witbrock, “AI for Complex Situations: Beyond Uniform Problem
Solving,” 2017.
[7] V. Vanhoucke and G. B. Robotics, “‘ OK Google , fold my laundry
s ’ il te plaı̂t ’.”
[8] T. Sunahase, Y. Baba, and H. Kashima, “Pairwise HITS : Quality
Estimation from Pairwise Comparisons in Creator-Evaluator Crowd-
sourcing Process,” Proceedings of the 31th Conference on Artificial
Intelligence (AAAI 2017), no. Kleinberg, pp. 977–983, 2017.
[9] J. Goldsmith and E. Burton, “Why teaching ethics to AI practitioners
is important,” 31st AAAI Conference on Artificial Intelligence, AAAI
2017, pp. 110–114, 2017.
[10] A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen,
“Combining deep learning with information retrieval to localize buggy
files for bug reports,” Proceedings - 2015 30th IEEE/ACM International
Conference on Automated Software Engineering, ASE 2015, pp. 476–
481, 2016.
[11] M. White, M. Tufano, C. Vendome, and D. Poshyvanyk, “Deep
learning code fragments for code clone detection,” Proceedings of
the 31st IEEE/ACM International Conference on Automated Software
Engineering - ASE 2016, pp. 87–98, 2016.
[12] X. Li, Y. Liang, H. Qian, Y.-Q. Hu, L. Bu, Y. Yu, X. Chen, and
X. Li, “Symbolic Execution of Complex Program Driven by Machine
Learning Based Constraint Solving,” Ase, pp. 554–559, 2016.
[13] N. Li, Y. Lei, H. R. Khan, J. Liu, and Y. Guo, “Applying combi-
natorial test data generation to big data applications,” Proceedings of
the 31st IEEE/ACM International Conference on Automated Software
Engineering - ASE 2016, pp. 637–647, 2016.
[14] W. Zhang, X. Cao, R. Wang, Y. Guo, and Z. Chen, “Binarized Mode
Seeking for Scalable Visual Pattern Discovery,” Cvpr2017, pp. 3864–
3872, 2017.
[15] J. Wu, J. B. Tenenbaum, and P. Kohli, “Neural Scene De-rendering,”
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pp. 699–707, 2017.
[16] W. Treible, P. Saponaro, S. Sorensen, A. Kolagunda, M. O. Neal,
B. Phelan, K. Sherbondy, and C. Kambhamettu, “CATS : A Color
and Thermal Stereo Benchmark,” Cvpr, pp. 134–142, 2017.
[17] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and
R. Webb, “Learning from Simulated and Unsupervised Images through
Adversarial Training,” pp. 2242–2251, 2016.
[18] K. Sasaki, S. Iizuka, E. Simo-serra, and H. Ishikawa, “Joint Gap
Detection and Inpainting of Line Drawings,” Cvpr, pp. 5768–5776,
2017.
[19] P. Sangkloy, J. Lu, C. Fang, F. Yu, and J. Hays, “Scribbler: Controlling
Deep Image Synthesis with Sketch and Color,” pp. 6836–6845, 2016.
282
[20] K. Nakamura, S. Yeung, A. Alahi, and L. Fei-Fei, “Jointly
Learning Energy Expenditures and Activities using Egocentric
Multimodal Signals,” Cvpr, pp. 6817–6826, 2017. [Online]. Available:
http://vision.stanford.edu/pdf/nakamura2017cvpr.pdf
[21] H. Le, T.-j. Chin, and D. Suter, “An Exact Penalty Method for Locally
Convergent Maximum Consensus,” Cvpr2017, pp. 1888–1896, 1888.
[22] I. Kokkinos, “UberNet: Training a ‘Universal’ Convolutional Neural
Network for Low-, Mid-, and High-Level Vision using Diverse Datasets
and Limited Memory,” 2016.
[23] W. Kehl, F. Tombari, S. Ilic, and N. Navab, “Real-Time 3D Model
Tracking in Color and Depth on a Single CPU Core,” 2017 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pp.
465–473, 2017.
[24] D. He, X. Yang, C. Liang, Z. Zhou, D. Kifer, C. L. Giles, and
A. Ororbia, “Multi-scale FCN with Instance Aware Segmentation for
Arbitrary Oriented Word Spotting In The Wild,” Cvpr 2017, no. 1,
2017.
[25] C. Fan, J. Lee, M. Xu, K. K. Singh, Y. J. Lee, D. J. Crandall, and
M. S. Ryoo, “Identifying First-person Camera Wearers in Third-person
Videos,” no. 1, pp. 4734–4742, 2017.
[26] Y. Chen and Y.-j. Liu, “Learning to Rank Retargeted Images,” Learn-
ing, pp. 3994–4002, 2017.
[27] L. Castrejn, K. Kundu, R. Urtasun, and S. Fidler, “Annotating object
instances with a polygon-rnn,” in 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), July 2017, pp. 4485–4493.
[28] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, “Network
dissection: Quantifying interpretability of deep visual representations,”
in 2017 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), July 2017, pp. 3319–3327.
[29] X. Alameda-Pineda, A. Pilzer, D. Xu, N. Sebe, and E. Ricci, “Vi-
raliency: Pooling local virality,” in 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), July 2017, pp. 484–492.
[30] V. K. Adhikarla, M. Vinkler, D. Sumin, R. K. Mantiuk, K. Myszkowski,
H. P. Seidel, and P. Didyk, “Towards a quality metric for dense light
fields,” in 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), July 2017, pp. 3720–3729.
[31] S. Bosse, “Distributed machine learning with self-organizing mobile
agents for earthquake monitoring,” Proceedings - IEEE 1st Interna-
tional Workshops on Foundations and Applications of Self-Systems,
FAS-W 2016, pp. 126–132, 2016.
[32] M. Ribeiro, K. Grolinger, and M. A. Capretz, “MLaaS:
Machine Learning as a Service,” 2015 IEEE 14th
International Conference on Machine Learning and Applications
(ICMLA), no. c, pp. 896–902, 2015. [Online]. Available:
http://ieeexplore.ieee.org/document/7424435/
[33] B. Chen, D. Kumor, and E. Bareinboim, “Identification and Model
Testing in Linear Structural Equation Models using Auxiliary
Variables,” Proceedings of the 34th International Conference on
Machine Learning, vol. 70, pp. 757–766, 2017. [Online]. Available:
http://proceedings.mlr.press/v70/chen17f.html
[34] M. Harman, P. McMinn, J. De Souza, and S. Yoo, “Search
based software engineering: Techniques, taxonomy, tutorial,”
Search, vol. 2012, pp. 1–59, 2011. [Online]. Available:
http://discovery.ucl.ac.uk/1340709/
[35] N. Erman, V. Tufvesson, M. Borg, P. Runeson, and A. Ardö, “Navigat-
ing information overload caused by automated testing - A clustering
approach in multi-branch development,” 2015 IEEE 8th International
Conference on Software Testing, Verification and Validation, ICST 2015
- Proceedings, 2015.
[36] R. Carbone, L. Compagna, A. Panichella, and S. E. Ponta, “Security
threat identification and testing,” 2015 IEEE 8th International Con-
ference on Software Testing, Verification and Validation, ICST 2015 -
Proceedings, 2015.
[37] S. F. Sun and A. Podgurski, “Properties of Effective Metrics for
Coverage-Based Statistical Fault Localization,” Proceedings - 2016
IEEE International Conference on Software Testing, Verification and
Validation, ICST 2016, pp. 124–134, 2016.
[38] K. Moran, M. Linares-Vasquez, C. Bernal-Cardenas, C. Vendome, and
D. Poshyvanyk, “Automatically Discovering, Reporting and Reproduc-
ing Android Application Crashes,” Proceedings - 2016 IEEE Inter-
national Conference on Software Testing, Verification and Validation,
ICST 2016, pp. 33–44, 2016.
[39] B. Marculescu, R. Feldt, and R. Torkar, “Using Exploration Focused
Techniques to Augment Search-Based Software Testing: An Experi-
mental Evaluation,” Proceedings - 2016 IEEE International Conference
283
on Software Testing, Verification and Validation, ICST 2016, pp. 69–79,
2016.
[40] M. Tappler, B. K. Aichernig, and R. Bloem, “Model-Based Testing IoT
Communication via Active Automata Learning,” Proceedings - 10th
IEEE International Conference on Software Testing, Verification and
Validation, ICST 2017, pp. 276–287, 2017.
[41] D. Pradhan, S. Wang, S. Ali, T. Yue, and M. Liaaen, “CBGA-
ES: A Cluster-Based Genetic Algorithm with Elitist Selection for
Supporting Multi-Objective Test Optimization,” Proceedings - 10th
IEEE International Conference on Software Testing, Verification and
Validation, ICST 2017, pp. 367–378, 2017.
[42] G. Gay, “The Fitness Function for the Job: Search-Based Generation
of Test Suites That Detect Real Faults,” Proceedings - 10th IEEE Inter-
national Conference on Software Testing, Verification and Validation,
ICST 2017, pp. 345–355, 2017.
[43] P. McMinn, “Search-Based Software Testing: Past, Present and
Future,” 2011 IEEE Fourth International Conference on Software
Testing, Verification and Validation Workshops, pp. 153–163, 2011.
[Online]. Available: http://ieeexplore.ieee.org/document/5954405/
[44] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A Survey
on Software Fault Localization,” IEEE Transactions on Software En-
gineering, vol. 42, no. 8, pp. 707–740, 2016.
[45] H. Narasimhan, S. Agarwal, and D. C. Parkes, “Automated mechanism
design without money via machine learning,” IJCAI International Joint
Conference on Artificial Intelligence, vol. 2016-Janua, pp. 433–439,
2016.
[46] Y. F. Li, S. B. Wang, and Z. H. Zhou, “Graph quality judgement:
A large margin expedition,” IJCAI International Joint Conference on
Artificial Intelligence, vol. 2016-Janua, pp. 1725–1731, 2016.
[47] N. Alechina, M. Dastani, and B. Logan, “Verifying existence of
resource-bounded coalition uniform strategies,” IJCAI International
Joint Conference on Artificial Intelligence, vol. 2016-Janua, pp. 24–
30, 2016.
[48] S. Adriaensen and A. Nowé, “Towards a white box approach to
automated algorithm design,” IJCAI International Joint Conference on
Artificial Intelligence, vol. 2016-Janua, pp. 554–560, 2016.
[49] V. Noroozi, L. Zheng, S. Bahaadini, S. Xie, and P. S. Yu, “SEVEN:
Deep SEmi-supervised verification networks,” IJCAI International
Joint Conference on Artificial Intelligence, pp. 2571–2577, 2017.
[50] P. Kouvaros and A. Lomuscio, “Verifying fault-tolerance in parame-
terised multi-agent systems,” IJCAI International Joint Conference on
Artificial Intelligence, pp. 288–294, 2017.
[51] J. Kong and A. Lomuscio, “Model checking multi-agent systems
against LDLK specifications,” IJCAI International Joint Conference
on Artificial Intelligence, pp. 1138–1144, 2017.
[52] N. Gorogiannis and F. Raimondi, “A Novel Symbolic Approach to
Verifying Epistemic Properties of Programs ,” pp. 206–212, 2009.
[53] F. Belardinelli, L. Ibisc, and I. Toulouse, “Parameterised Verification
of Data-aware Multi-agent Systems,” pp. 98–104, 2016.
[54] F. Belardinelli, L. Ibisc, A. Murano, and S. Rubin, “Verification of
Broadcasting Multi-Agent Systems against an Epistemic Strategy Logic
Imperial College London,” pp. 91–97, 2014.
[55] O. Tripp, O. Weisman, and L. Guy, “Finding Your Way in the
Testing Jungle: A Learning Approach to Web Security Testing,”
Proceedings of the 2013 International Symposium on Software
Testing and Analysis, pp. 347–357, 2013. [Online]. Available:
http://doi.acm.org/10.1145/2483760.2483776
[56] F. M. Kifetew, A. Panichella, A. De Lucia, R. Oliveto, and P. Tonella,
“Orthogonal exploration of the search space in evolutionary test case
generation,” Proceedings of the 2013 International Symposium on
Software Testing and Analysis - ISSTA 2013, p. 257, 2013. [Online].
Available: http://dl.acm.org/citation.cfm?doid=2483760.2483789
[57] F. Howar, D. Giannakopoulou, and Z. Rakamarić, “Hybrid learning:
interface generation through static, dynamic, and symbolic analysis,”
Proceedings of the 2013 International Symposium on Software Testing
and Analysis - ISSTA 2013, p. 268, 2013. [Online]. Available:
http://dl.acm.org/citation.cfm?doid=2483760.2483783
[58] I. Medeiros, N. Neves, and M. Correia, “DEKANT: A Static
Analysis Tool that Learns to Detect Web Application Vulnerabilities,”
Proceedings of the 14th ACM conference on Computer and
communications security CCS 07, p. 529, 2007. [Online]. Available:
http://portal.acm.org/citation.cfm?doid=1315245.1315311
[59] T.-D. B. Le, D. Lo, C. Le Goues, and L. Grunske, “A learning-to-rank
based fault localization approach using likely invariants,” Proceedings
of the 25th International Symposium on Software Testing and
 Analysis - ISSTA 2016, pp. 177–188, 2016. [Online]. Available:
http://dl.acm.org/citation.cfm?doid=2931037.2931049
[60] H. Spieker, A. Gotlieb, D. Marijan, and M. Mossige, “Reinforcement
Learning for Automatic Test Case Prioritization and Selection in
Continuous Integration,” Proceedings of 26th International Symposium
on Software Testing and Analysis (ISSTA’17), pp. 12—-22, 2017.
[61] M. Santolucito, “Version Space Learning for Verification on Temporal
Differentials,” Proceedings of the 26th ACM SIGSOFT International
Symposium on Software Testing and Analysis, pp. 428–431, 2017.
[Online]. Available: http://doi.acm.org/10.1145/3092703.3098238
[62] D. S. Katz, “Understanding Intended Behavior using Models of Low-
Level Signals,” pp. 424–427.
[63] J. Hotzkow, “Automatically Inferring and Enforcing User
Expectations,” Proceedings of the 26th ACM SIGSOFT
International Symposium on Software Testing and Analysis -
ISSTA 2017, no. July, pp. 420–423, 2017. [Online]. Available:
http://dl.acm.org/citation.cfm?doid=3092703.3098236
[64] K. R. Varshney, “Engineering safety in machine learning,” 2016 Infor-
mation Theory and Applications Workshop, ITA 2016, 2017.
[65] G. I. Webb and F. Petitjean, “A Multiple Test Correction for Streams
and Cascades of Statistical Hypothesis Tests,” Proceedings of the 22nd
ACM SIGKDD International Conference on Knowledge Discovery and
Data Mining - KDD ’16, pp. 1255–1264, 2016. [Online]. Available:
http://dl.acm.org/citation.cfm?doid=2939672.2939775
[66] M. T. Ribeiro, S. Singh, and C. Guestrin, “”why should i
trust you?”: Explaining the predictions of any classifier,” in
Proceedings of the 22Nd ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining, ser. KDD ’16. New
York, NY, USA: ACM, 2016, pp. 1135–1144. [Online]. Available:
http://doi.acm.org/10.1145/2939672.2939778
[67] R. Parekh, “Designing AI at Scale to Power Everyday Life,” Proceed-
ings of the 23rd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining - KDD ’17, pp. 27–27, 2017. [Online].
Available: http://dl.acm.org/citation.cfm?doid=3097983.3105815
[68] D. Baylor and E. Breck, “TFX: A TensorFlow-Based Production-Scale
Machine Learning Platform,” Kdd, pp. 1387–1395, 2017.
[69] J. Van Der Herten, I. Couckuyt, D. Deschrijver, P. Demeester, and
T. Dhaene, “Adaptive modeling and sampling methodologies for In-
ternet of Things applications,” Proceedings of the 18th Mediterranean
Electrotechnical Conference: Intelligent and Efficient Technologies and
Services for the Citizen, MELECON 2016, no. April, pp. 18–20, 2016.
[70] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner,
V. Chaudhary, M. Young, and D. Dennison, “Hidden Technical
Debt in Machine Learning Systems,” Nips, pp. 2494–2502, 2015.
[Online]. Available: http://papers.nips.cc/paper/5656-hidden-technical-
debt-in-machine-learning-systems.pdf
[71] F. Yang, A. Ramdas, K. Jamieson, and M. J. Wainwright, “A framework
for Multi-A(rmed)/B(andit) testing with online FDR control,” no. 3,
2017. [Online]. Available: http://arxiv.org/abs/1706.05378
[72] R. Sen, A. T. Suresh, K. Shanmugam, A. G. Dimakis, and S. Shakkot-
tai, “Model-Powered Conditional Independence Test,” no. Nips, pp.
1–11, 2017.
[73] P. Schulam and S. Saria, “Reliable Decision Support using
Counterfactual Models,” no. Nips, 2017. [Online]. Available:
http://arxiv.org/abs/1703.10651
[74] J. Z. Liu and B. Coull, “Robust Hypothesis Test for Nonlinear Effect
with Gaussian Processes,” Nips2017, vol. 2, no. Nips, pp. 1–10, 2017.
[Online]. Available: http://arxiv.org/abs/1710.01406
[75] H. C. L. Law, C. Yau, and D. Sejdinovic, “Testing and Learning
on Distributions with Symmetric Noise Invariance,” no. Mmd, 2017.
[Online]. Available: http://arxiv.org/abs/1703.07596
[76] W. Jitkrittum, W. Xu, Z. Szabo, K. Fukumizu, and A. Gretton, “A
Linear-Time Kernel Goodness-of-Fit Test,” no. Nips, pp. 0–3, 2017.
[Online]. Available: http://arxiv.org/abs/1705.07673
[77] Z. Daniel Guo, P. S. Thomas, and E. Brunskill, “Using
Options and Covariance Testing for Long Horizon Off-
Policy Policy Evaluation,” Advances in Neural Information
Processing Systems 30 (NIPS 2017), no. Nips, 2017. [On-
line]. Available: http://papers.nips.cc/paper/6843-using-options-and-
covariance-testing-for-long-horizon-off-policy-policy-evaluation.pdf
[78] F. Cecchi and N. Hegde, “Adaptive Active Hypothesis Testing under
Limited Information,” no. d, pp. 1–9, 2017.
[79] H. Assem, L. Xu, T. S. Buda, and D. O’Sullivan, “Machine learning
as a service for enabling Internet of Things and People,” Personal and
Ubiquitous Computing, vol. 20, no. 6, pp. 899–914, 2016.
284
[80] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated
whitebox testing of deep learning systems,” in Proceedings of the
26th Symposium on Operating Systems Principles, ser. SOSP ’17.
New York, NY, USA: ACM, 2017, pp. 1–18. [Online]. Available:
http://doi.acm.org/10.1145/3132747.3132785
[81] Amazon, “Amazon Alexa.” [Online]. Available:
https://developer.amazon.com/alexa
[82] IBM, “With Watson Program,” 2018. [Online]. Available:
https://www.ibm.com/watson/with-watson/
[83] R. G. Smith, “On the development of commercial expert systems,”
AI Magazine, vol. 5, no. 3, pp. 61–73, 1984. [Online]. Available:
https://www.aaai.org/ojs/index.php/aimagazine/article/view/449
[84] S. Sievers, M. Ortlieb, and M. Helmert, “Efficient Implementation of
Pattern Database Heuristics for Classical Planning,” Symposium on
Combinatorial Search, pp. 105–111, 2012.
[85] A. Ramaswamy, B. Monsuez, and A. Tapus, “AI Dimensions in
Software Development for Human-Robot Interaction Systems,” pp.
128–130, 2014.
[86] D. S. Prerau, “Knowledge Acquisition in the Development of a Large
Expert System,” AI Magazine, vol. 8, no. 2, pp. 43–51, 1987.
[87] G. Peter, “Knowledge-Based Software Engineering Conference,” 1992.
[88] M. R. Lowry, “Software Engineering in the Twenty-First Century,” AI
Magazine, vol. 13, no. 3, pp. 71–87, 1992.
[89] S. Giroux and N. Bier, “Cognitive Assistance to Meal Preparation:
Design, Implementation, and Assessment in a Living Lab,”
2015 AAAI Spring . . . , pp. 01–25, 2015. [Online]. Available:
https://www.aaai.org/ocs/index.php/SSS/SSS15/paper/view/10329/10013
[90] M. A. Cohen, F. E. Ritter, and S. R. Haynes, “Applying Software
Engineering to Agent Development,” pp. 25–44, 2010.
[91] S. Burton, K. Swanson, and L. Leonard, “Quality and Knowledge in
Software Engineering,” AI Magazine, vol. 14, no. 4, pp. 43–50, 1993.
[92] R. Hollander, “Alibaba, Baidu, and Tencent, China’s powerhouses,
focus on AI to surpass the US - Business Insider,” 2017. [Online].
Available: http://www.businessinsider.com/alibaba-baidu-and-tencent-
chinas-powerhouses-focus-on-ai-to-surpass-the-us-2017-11
[93] A. Nordrum, “Automatic Speaker Verification Systems Can Be Fooled
by Disguising Your Voice - IEEE Spectrum,” 2017. [Online]. Avail-
able: https://spectrum.ieee.org/tech-talk/telecom/security/automatic-
speaker-verification-systems-can-be-fooled-by-disguising-your-voice
[94] S. K. Moore, “DARPA Seeking AI That Learns
All the Time - IEEE Spectrum,” 2017. [Online].
Available: https://spectrum.ieee.org/cars-that-think/robotics/artificial-
intelligence/darpa-seeking-ai-that-can-learn-all-the-time
[95] J. Hsu, “Deep Learning AI for NASA Powers Earth
Robots - IEEE Spectrum,” 2017. [Online]. Available:
https://spectrum.ieee.org/automaton/robotics/artificial-intelligence/ai-
startup-neurala-deep-learning-for-nasa-powers-earth-robots
[96] ——, “A New Way to Find Bugs in Self-Driving
AI Could Save Lives - IEEE Spectrum,” 2017. [On-
line]. Available: https://spectrum.ieee.org/tech-talk/robotics/artificial-
intelligence/better-bug-hunts-in-selfdriving-car-ai-could-save-lives
[97] E. Ackerman, “After Mastering Singapore’s Streets, NuTonomy’s
Robo-taxis Are Poised to Take on New Cities - IEEE Spectrum,”
2016. [Online]. Available: https://spectrum.ieee.org/transportation/self-
driving/after-mastering-singapores-streets-nutonomys-robotaxis-are-
poised-to-take-on-new-cities
[98] N. Papernot, M. Abadi, Ú. Erlingsson, I. Goodfellow, and K. Talwar,
“Semi-supervised Knowledge Transfer for Deep Learning from Private
Training Data,” no. 2015, pp. 1–16, 2016. [Online]. Available:
http://arxiv.org/abs/1610.05755
[99] T. Miyato, A. M. Dai, and I. Goodfellow, “Adversarial Training
Methods for Semi-Supervised Text Classification,” pp. 1–11, 2016.
[Online]. Available: http://arxiv.org/abs/1605.07725
[100] S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel,
“Adversarial Attacks on Neural Network Policies,” 2017. [Online].
Available: http://arxiv.org/abs/1702.02284
[101] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and
D. Amodei, “Deep reinforcement learning from human preferences,”
2017. [Online]. Available: http://arxiv.org/abs/1706.03741
[102] V. Cheung, J. Schneider, I. Sutskever, and G. Brockman,
“Infrastructure for Deep Learning,” pp. 1–11, 2016. [Online].
Available: https://openai.com/blog/infrastructure-for-deep-learning/
[103] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and
D. Mané, “Concrete Problems in AI Safety,” pp. 1–29, 2016.
 1

Software Engineers vs.

Machine Learning Algorithms:
An Empirical Study Assessing
Performance and Reuse Tasks
Nathalia Nascimento, Carlos Lucena, Paulo Alencar and Donald Cowan

Abstract—Several papers have recently contained reports on applying machine learning (ML) to the
arXiv:1802.01096v2 [cs.SE] 7 Feb 2018

automation of software engineering (SE) tasks, such as project management, modeling and development.
However, there appear to be no approaches comparing how software engineers fare against machine-learning
algorithms as applied to specific software development tasks. Such a comparison is essential to gain insight
into which tasks are better performed by humans and which by machine learning and how cooperative work or
human-in-the-loop processes can be implemented more effectively. In this paper, we present an empirical study
that compares how software engineers and machine-learning algorithms perform and reuse tasks. The
empirical study involves the synthesis of the control structure of an autonomous streetlight application. Our
approach consists of four steps. First, we solved the problem using machine learning to determine specific
performance and reuse tasks. Second, we asked software engineers with different domain knowledge levels to
provide a solution to the same tasks. Third, we compared how software engineers fare against
machine-learning algorithms when accomplishing the performance and reuse tasks based on criteria such as
energy consumption and safety. Finally, we analyzed the results to understand which tasks are better
performed by either humans or algorithms so that they can work together more effectively. Such an
understanding and the resulting human-in-the-loop approaches, which take into account the strengths and
weaknesses of humans and machine-learning algorithms, are fundamental not only to provide a basis for
cooperative work in support of software engineering, but also, in other areas.

Index Terms—Machine learning, human-in-the-loop, software engineer, automatic software engineering,

internet of things, empirical study

1 I NTRODUCTION

S OFTWARE engineering processes can be very

complex, costly and time-consuming [1].
They typically consist of a collection of related
tasks [2] such as designing, implementing, main-
taining, testing and reusing software applications
[3]. In addition, as software has become embed-
ded in systems of all kinds, millions of computer
programs have to be corrected, adapted, and
• Laboratory of Software Engineering (LES) at the Depart-
ment of Informatics, Pontifical Catholic University of Rio de
Janeiro, Brazil. CAPES scholarship/Program 194/Process:
88881.134630/2016-01
E-mail: nnascimento,lucena@inf.puc-rio.br
• See http://www.inf.puc-rio.br/∼nnascimento/
• David R. Cheriton School of Computer Science. University
of Waterloo, Canada.
E-mail: palencar,dcowan@csg.uwaterloo.ca
enhanced [2]. As a result, the field of software
engineering requires millions of skilled Infor-
mation Technology (IT) professionals to create
millions of lines of code, which must be installed,
configured, tuned, and maintained. According to
Kephart (2005) [4], in the near future, it will be ex-
tremely challenging to manage IT environments,
even for the most skilled IT professionals.
Several researchers have proposed the use of
artificial intelligence, especially machine-learning
(ML) techniques, to automate different software
engineering (SE) tasks [3], [5]–[20]. For exam-
ple, Zhang has extensively studied this theme
recently and in [3] he stated that:
”The field of software engineering turns
out to be a fertile ground where many
software development and maintenance

tasks could be formulated as learning
problems and approached in terms of
learning algorithms.”
However, there is a lack of approaches to
compare how software engineers fare against
machine-learning algorithms for specific soft-
ware development tasks. This comparison is crit-
ical in order to evaluate which S.E. tasks are
better performed by automation and which re-
quire human involvement or human-in-the-loop
approaches [21], [22]. In practice, because there
are no explicit comparisons between the tasks
performed by engineers and automated proce-
dures, including machine learning, it is often not
clear when to use automation in a specific setting.
For example, a Brazilian company acquired a
software system to select petroleum exploration
models automatically, but the engineers decided
they could provide a better solution manually.
However, when there was a comparison of the
manual solution with the one provided automat-
ically by the system, it became clear that the
automated solution was better. This illustrates
that a lack of comparisons makes choosing a
manual or an automated solution or a combined
human-in-the-loop approach difficult.
This paper, contains an empirical study [23] to
compare how software engineers and machine-
learning algorithms achieve performance and
reuse tasks. The empirical study uses a case study
involving the creation of a control structure for
an autonomous streetlight application. The ap-
proach consists of four steps. First, the problem
was solved using machine learning to achieve
specific performance and reuse of tasks. Sec-
ond, we asked software engineers with different
domain-knowledge levels to provide a solution
to achieve the same tasks. Third, we compared
how software engineers compare with machine-
learning algorithms when accomplishing the per-
formance and reuse tasks based on criteria such
as energy consumption and safety. Finally, the
results were analyzed to understand which tasks
are better performed by either humans or al-
gorithms so that they can work together more
effectively.
Such an understanding is essential in re-
alizing novel human-in-the-loop approaches in
which machine-learning procedures assist soft-
ware developers in achieving tasks. Such human-
in-the-loop approaches, which take into account
the strengths and weaknesses of humans and
machine-learning algorithms, are fundamental
not only to provide a basis for cooperative work
2
in software engineering, but also in other appli-
cation areas.
This paper is organized as follows: Section
2 presents the empirical study describing re-
search questions, hypotheses and the objective
of the study. Section 3 presents the method se-
lected to collect our empirical data. Sections 4
and 5 present the experimental results. Section
6 presents the threats to the validity of our ex-
periment. Section 7 presents the related work.
The paper ends with concluding remarks and
suggestions for future work.
1.1 Motivation
The theme of this paper, namely whether arti-
ficial intelligence such as machine learning, can
benefit software engineering, has been investi-
gated since 1986, when Hebert A Simon pub-
lished a paper entitled “Whether software engi-
neering needs to be artificially intelligent” [24].
In this paper, Simon discussed “the roles that
humans now play versus the roles that could be
taken over by artificial intelligence in developing
computer systems.” Notwithstanding, in 1993,
Ian Sommerville raised the following question
[25]: “What of the future - can Artificial Intelli-
gence make a contribution to system engineer-
ing?” In this paper [25], Sommervile performed a
literature review in applications of artificial intel-
ligence to software engineering, and concluded
that:
“the contribution of AI will be in sup-
porting...activities that are characterized
by solutions to problems which are
neither right nor wrong but which are
more or less appropriate for a par-
ticular situation...For example, require-
ments specification and analysis which
involves extensive consultation with do-
main experts and in project manage-
ment.”
Several papers have since investigated the use
of Machine Learning (ML) [26] in solving differ-
ent software engineering (SE) tasks [5]–[20], [27]–
[103]. These investigations include approaches
to: i) project management [27]–[49], dealing with
problems related to cost, time, quality prediction,
and resource management; ii) defect prediction
[50]–[84]; iii) requirements management, focus-
ing on problems of classifying or representing re-
quirements [85]–[88], or generating requirements
[89]; iv) software development, such as code gen-
eration [20], [68], [90]–[96], synthesis [97]–[101],
and code evaluation [102], [103].

Most of these papers present successful ap-
plications of machine learning in software engi-
neering, showing that ML techniques can provide
correct automatic solutions to some SE problems.
However, very few papers discuss whether or
not a domain expert could propose a manual
solution more appropriate for the particular
situation. “More appropriate”, means a solution
that provides better performance or increases an-
other quality that is important to a particular ap-
plication scenario, such as user preference [104].
For example, in the medical and aviation engi-
neering fields, trust [105] in a solution provided
to the end-user is an important factor to consider
for a solution to be more appropriate. However,
although many authors [106]–[109] have been
promoting the use of neural networks [110] in
medicine, Abbas et al. [105] and Castelvecchi
[111] are among the few authors who questioned:
“what is the trustworthiness of a prediction made
by an artificial neural network?”
In other application scenarios, such as many
of those related to the Internet of Things (IoT)
[112], [113], numerous authors [93], [95], [101],
[114] consider the reuse of a solution as an im-
portant quality. They agree that to achieve the
goal of billions of things connected to the Internet
over the next few years [112], it is necessary to
find ways to reduce time to market. For example,
it is desirable that the solution or parts of the
solution to design autonomous streetlights [96]
for a specific scenario could be reused to design
streetlights for another scenario.
In particular, the Internet of Things has con-
siderably increased the number of approaches
that propose the use of machine learning to
automate software development [93]–[96], [101],
[115]. None of this research contains a compar-
ison of their results to experiments designed by
IoT experts. For example, do Nascimento and Lu-
cena [101], [116] developed a hybrid framework
that uses learning-based and manual program
synthesis for the Internet of Things (FIoT). They
generated four instances of the framework [101],
[108], [117], [118] and used learning techniques
to synthesize the control structure automatically.
These authors stated that the use of machine
learning made feasible the development of these
applications. However, they did not present any
experiment without using learning techniques. In
contrast, most of the solutions released for the
Internet of Things, such as Apple’s HomeKit’s
approach [119] and Samsung Smart Things, [120]
consider a software developer synthesizing the
3
control structure for each thing manually.
1.2 Objective
In this context, we decided to ask the following
question: “How do software engineers compare
with machine-learning algorithms?” To explore
this question, we selected the Internet of Things
as our application domain and then, compared
a solution provided by a skilled IoT professional
with a solution provided by a learning algorithm
with respect to performance and reuse tasks.
In short, Figure 1 depicts the theory [121] that
we investigate in this paper. According to the
theory, the variables that we intend to isolate
and measure are the performance and reusabil-
ity achieved from three kinds of solutions: i)
solutions provided by learning techniques; ii)
solutions provided by software engineers with
IoT skills; and iii) solutions provided by software
engineers without IoT skills.
To evaluate the relationship among these vari-
ables, we performed an empirical study, using
FIoT [101]. As shown in Figure 1, we raised
four research questions (RQx) to investigate our
theory’s propositions (e.g hypotheses (H-RQx)).
We present these questions and hypotheses in
Section 2. To collect and analyze our empiri-
cal data, we performed a controlled experiment.
To perform this experiment, we reproduced the
problem of synthesizing the control structure
of autonomous streetlights using neuroevolution
(i.e. “a learning algorithm which uses genetic
algorithms to train neural networks” [122]) pre-
sented in [118]. Then, we invited 14 software
engineers to provide a solution for the same
problem using the same architecture and envi-
ronment. Lastly, we compared the solution pro-
vided by the learning algorithm against the so-
lutions provided by the software engineers. In
this application of autonomous streetlights, we
are considering a “more appropriate” solution
as one that presents a better performance in
the main scenario [118] or can be satisfactorily
reused in a new scenario, based on criteria such
as minimal energy consumption and safety
(that is, maximum visual comfort in illuminated
areas).
 4

Technology SE Tasks
Actor
Synthesize and reuse
Machine Learning the control structure of
Techniques autonomous things

Software Engineers produce solutions

better than

with IoT skills

(H-RQ1, H-RQ2) Software System

produce solutions better Internet of Things

than (H-RQ3, H-RQ4)
without IoT skills Improve (H-RQ1, H-RQ3)
Performance
Increase (H-RQ2, H-RQ4)
Reusability

Fig. 1. Theory [121]: Machine Learning can create solutions more appropriate than software engineers in the context of
the Internet of Things.

2 H OW DO SOFTWARE ENGINEERS COM - RQ4. ...software engineers without IoT

PARE WITH MACHINE - LEARNING ALGO -
RITHMS ? A N E MPIRICAL S TUDY A D -
DRESSING P ERFORMANCE AND R EUSE IN
THE I OT D OMAIN
The experimental goal, based on the Goal-
Question-Metric (GQM) method [123] is to use
the Framework for the Internet of Things (FIoT)
for the purpose of comparing the use of an
automated approach against a manual approach
when synthesizing the control of autonomous
things with respect to their performance and
reuse.
For this purpose, we asked four research
questions (RQs) and performed a controlled ex-
periment [23] (section 3) to investigate them.
2.1 Questions
In terms of synthesizing the control structure of
autonomous things, how does the result from
a machine learning-based solution differ from
solutions provided by...
RQ1. ...software engineers with IoT
skills with respect to their perfor-
mance?
RQ2. ...software engineers with IoT
skills with respect to their re-usability?
RQ3. ...software engineers without IoT
skills with respect to their perfor-
mance?
skills with respect to their re-usability?
2.2 Hypotheses
Each RQ is based on one or more hypotheses,
which are described next.
H - RQ1.
• H0. An ML-based approach does not im-
prove the performance of autonomous
things compared to solutions provided by
IoT expert software engineers.
• HA. An ML-based approach improves the
performance of autonomous things com-
pared to solutions provided by IoT expert
software engineers.
H - RQ2.
• H0. An ML-based approach does not in-
crease the reuse of autonomous things
compared to solutions provided by IoT
expert software engineers.
• HA. An ML-based approach increases the
reuse of autonomous things compared to
solutions provided by IoT expert software
engineers.
H - RQ3.
• H0. An ML-based approach does not im-
prove the performance of autonomous
things compared to solutions provided by
software engineers without experience in
IoT development.
 5

• HA. An ML-based approach improves the TABLE 1

performance of autonomous things com-
pared to solutions provided by software
engineers without experience in IoT de-
velopment.
H - RQ4.
• H0. An ML-based approach does not in-
crease the reuse of autonomous things
compared to solutions provided by soft-
ware engineers without experience in IoT
development.
• HA. An ML-based approach increases the
reuse of autonomous things compared to
solutions provided by software engineers
without experience in IoT development.
Implementing FIoT flexible points to synthesize streetlight
controllers using a ML-based approach.
FIoT Framework Light Control Application
Controller Three Layer Neural Network
Collective Fitness Evaluation:
the solution is evaluated
based on the energy
consumption, the number of
Making Evaluation people that finished their
routes after the
simulation ends, and the
total time spent by people
to move during their trip
Evolutionary Algorithm:
Controller Adaptation Generate a pool of
at design time candidates to represent the
neural network parameters

2.3 The object of the study: The Framework TABLE 2

for the Internet of Things (FIoT)
The Framework for the Internet of Things (FIoT)
[101] is a hybrid software framework that allows
the developer to generate controller structures for
autonomous things through learning or procedu-
ral algorithms.
If a researcher develops an application using
FIoT, the application will contain a Java software
component already equipped with modules for
detecting autonomous things in an environment,
assigning a controller to a specific thing, creating
software agents, collecting data from devices and
supporting the communication structure among
agents and devices.
Some features are variable and may be select-
ed/developed according to the application type,
as follows: (i) a control module such as “if-else”,
neural network or finite state machine; (2) an
adaptive technique to synthesize the controller
at design-time, such as reinforcement learning
[124] or genetic algorithm; and (iii) an evaluation
process to evaluate the behavior of autonomous
things that are making decisions based on the
controller.
For example, Table 1 summarizes how the
“Streetlight Control” application will adhere
to the proposed framework using a machine
learning-based approach, while extending the
FIoT flexible points.
Table 2 summarizes how the “Streetlight Con-
trol” application will adhere to the proposed
framework using a solution provided by a soft-
ware engineer, while extending the FIoT flexible
points.
Our goal is to provide both solutions to the
same application and compare the results based
on the same evaluation process.
Implementing FIoT flexible points to synthesize streetlight
controllers using a solution provided by a Software
Engineer
FIoT Framework Light Control Application
if-else module provided by a
Controller
software engineer
Collective Fitness Evaluation:
the solution is evaluated
based on the energy
consumption, the number of
Making Evaluation people that finished their
routes after the
simulation ends, and the
total time spent by people
to move during their trip
Controller Adaptation
None
at design time
3 C ONTROLLED E XPERIMENT
The first step of the experiment was to reproduce
the experiment presented in [118] by using a not
supervised learning approach. Then, we invited
14 software engineers to provide a solution for
the same problem. Finally, we compared the so-
lution provided through the learning algorithm
against solutions provided by the participants.
3.1 Participant Analysis
As we have described previously, the knowl-
edge in the application domain is an important
variable in our empirical study. Therefore, before
performing the controlled experiment, we asked
participants to describe their experience with the
development of applications based on the In-
ternet of Things, that is, developing distributed
systems with embedded characteristics, such as
providing each element of the system with sen-
sors and actuators. As shown in Figure 2, 43% of
 6

Medium Each streetlight has to execute three tasks

High
14%
Low 22%
21%
43%
None
Fig. 2. Experience of participants in developing applica-
tions based on the Internet of Things.
participants have never developed an application
based on the Internet of Things and 57% have
developed at least one application.
3.2 Experiment: Streetlight Application
In short, our experiment involves developing
autonomous streetlights. The overall goal of this
application is to reduce the energy consump-
tion while maintaining appropriate visibility in
illuminated areas [118]. For this purpose, we
provided each streetlight with ambient bright-
ness and motion sensors, and an actuator to
control light intensity. In addition, we also pro-
vided streetlights with wireless communicators
as shown in Figure 3. Therefore, the streetlights
are able to cooperate with each other to establish
the most likely routes for passers-by and thus
achieve the goal of minimizing energy consump-
tion.
every second: data collection, decision-making
and action enforcement. The first task consists
of receiving data related to people flow, ambi-
ent brightness, data from the neighboring street-
lights and current light status (activation level
of sensors and the previous output value of
listeningDecision). The second task consists of
analyzing collected data and making decisions
about actions to be enforced. The last task is
the action enforcement, which consists of setting
the value of three output variables: (i) listen-
ingDecision, that enables the streetlight to receive
signals from neighboring streetlights in the next
cycle; (ii) wirelessTransmitter, a signal value to
be transmitted to neighboring streetlights; and
(iii) lightDecision, that activates the light’s OF-
F/DIM/ON functions.
The interested reader may consult a more
extensive paper about the application scenario
[118] 1 .
3.2.1 The Challenge
As we explained to the participants, the tasks
of collecting data and enforcing actions have
already been implemented. The challenge was
to provide a solution for the task of making
decisions, as depicted in Figure 4.

Lighting
(Dark/DIM/Light) Light Decision
(OFF/DIM/ON)
Presence Fig. 4. The challenge: how does a streetlight make deci-
(Yes/No) sions based on collected data?
Wireless
Data collected from Transmitter
the closest street light (0.0/0.5/1.0) We provided pseudocode that considered all
(0.0/0.5/1.0) possible combinations of input variables. Then,
Listening participants decided how to set output variables
Previous Listening Decision
Decision
(Yes/No)
according to the collected data. Part 2 of this
(Yes/No)
pseudocode is depicted in Figure 5.
Each participant provided a different solu-
Fig. 3. Variables collected and set by streetlights. tion. Therefore, we conducted the experiment by

Each streetlight in the simulation has a mi-
crocontroller that is used to detect the proximity
of a person, and control the closest streetlight. A
streetlight can change the status of its light to ON,
OFF or DIM.
1. All documents that we prepared to explain this appli-
cation scenario to participants are available at
http://www.inf.puc-rio.br/.̃nnascimento/projects.html
2. The pseudocode that we provided to participants is
available at:
http://www.inf.puc-rio.br/.̃nnascimento/projects.html
 7

Fig. 5. Small portion of the pseudocode of the decision module that was filled by participants.

if (lighting_sensor = Medium AND detected_person =

NO AND data_collected = 0.0 AND
previous_listening_decision = YES) then {
light_decision = __X_OFF____DIM____ON
wireless_transmitter = __X_0.0 ___0.5____1.0
listening_decision = _X__YES ____NO
} if (lighting_sensor = Medium AND detected_person =
YES AND data_collected = 0.0 AND
previous_listening_decision = YES) then {
light_decision = ___OFF__X__DIM____ON
wireless_transmitter = _X__0.0 ___0.5____1.0
listening_decision = __X_YES ____NO
} if (lighting_sensor = Dark AND detected_person = NO
AND data_collected = 0.0 AND
previous_listening_decision = NO) then {
light_decision = __X_OFF____DIM____ON
wireless_transmitter = ___0.0 __X_0.5____1.0
listening_decision = ___YES __X__NO
}

Fig. 6. Small portion of the rule decisions that was synthesized according to the learning-based approach.

using each one. In addition, we also considered
a “zeroed” solution, which always sets all values
to zero. This zeroed solution is supposed to be
the worst solution, since streetlights will always
switch their lights to OFF.
3.2.2 The solution generated by a machine-
learning algorithm
We compared the results from all of these ap-
proaches to the result produced using the ma-
chine learning approach. As do Nascimento and
Lucena explain in [118], the learning approach
uses a three-layer feedforward neural network
combined with an evolutionary algorithm to gen-
erate decision rules automatically. Figure 6 de-
picts some of the rules that were generated by the
evolved neural network. The interested reader
can consult more extensive papers [101], [118] or
read Nascimento’s dissertation [116] (chap. ii, sec.
iii).
Based on the generated rules and the system
execution, we observe that using the solution
provided by the neural network, only the street-
lights with broken lamps emit “0.5” from their
wireless transmitters.
In addition, we also observed that a streetlight
that is not broken switches its lamp ON if it de-
tects a persons proximity or receives “0.5” from a
wireless transmitter.
3.2.3 Scenario constraints
Before starting a solution, each participant
should consider the following constraints:
• Do not take light numbering into account,
since your solution may be used in differ-
ent scenarios (see an example of a scenario
in Figure 7).
• Three streetlights will go dark during the
simulation.
• People walk along different paths starting
at random departure points. Their role is
to complete their routes by reaching a des-
tination point. The number of people that
finished their routes after the simulation
ends, and the total time spent by people
 8

moving during their trip are the most Figure 7 depicts the elements that are part of the
important factors for a good solution. application namely, streetlights, people, nodes
• A person can only move if his current and and edges.
next positions are not completely dark.
In addition, we also consider that people
walk slowly if the place is partially devoid
of light. Execution: 12 seconds
• The energy consumption also influences - A person moves from
the solution evaluation. one point to another in
one second or a second
• The energy consumption is proportional and a half.
- Street lights execute
to the light status (OFF/DIM/ON). cycles of 1 second
• We also consider the use of the wireless
ON
transmitter to calculate energy consump- DIM
OFF
tion (if the streetlight emits something Broken Lamps

different from “0.0”, it consumes 0.1 of departure points

target points
energy).
Therefore, each solution is evaluated after the
Fig. 7. Simulated Neighborhood.
simulation ends based on the energy consump-
tion, the number of people that finished their Nascimento and Lucena [118] modeled the
routes after the simulation ends, and the total scenario as a graph, in which a node represents
time spent by people moving during their trip. a streetlight position and an edge represents
the smallest distance between two streetlights.
(completedP eople × 100) The graph representing the streetlight network
pP eople = (1)
totalP eople consists of 18 nodes and 34 edges. Each node
(totalEnergy × 100) represents a streetlight. In the graph, the yellow,
pEnergy = 11×(timeSimulation×totalSmartLights) gray, black and red triangles represent the street-
( 10 ) light status (ON/DIM/OFF/Broken Lamp). Each
(2) edge is two-way and links two nodes. In addi-
(totalT imeT rip × 100)
pT rip = 3×timeSimulation tion, each edge has a light intensity parameter
(( (2) ) × totalP eople) that is the sum of the environmental light and the
(3) brightness from the streetlights in its nodes. Their
f itness = (1.0 × pP eople) − (0.6 × pT rip)− goal is to simulate different lighting in different
(0.4 × pEnergy) neighborhood areas.
(4)
Equations (1) through (4) show the values to
be calculated for the evaluation in which pP eople
is the percentage of the number of people that
completed their routes before the end of the 1 second/1 second and a half
simulation out of the total number of people
in the simulation; pEnergy is the percentage of
energy that was consumed by streetlights out of
the maximum energy value that could be con-
sumed during the simulation. We also considered
the use of the wireless transmitter to calculate
energy consumption; pT rip is the percentage of
the total duration time of peoples trips out of the
maximum time value that their trip could spend;
and f itness is the fitness of each candidate that Fig. 8. Person moving in the simulated Neighborhood.
encodes the proposed solution.
As depicted in Figure 8, only one person was
3.2.4 Example - Simulating the environment started in the scenario that we showed to partic-
We showed participants the same simulated ipants. For instance, the person starting at point
neighborhood scenario that was used by the ge- 0 has point 11 as a target. We ask participants to
netic algorithm to evolve the neural network. provide a solution to streetlights to assure that

this person will conclude his route before the
simulation ends after 12 seconds.
3.2.5 New Scenario: Unknown environment
The second step of the experiment consists of
executing solutions from participants and the
learning approach in a new scenario, but with the
same constraints. This scenario, that is depicted
in Figure 9 was not used by the learning algo-
rithm and was not presented to participants.
The goal of this new part of the experiment
is to verify if the decision module that was de-
signed to control streetlights in the first scenario
can be reused in another scenario.
Fig. 9. Simulating a new neighborhood.
In this new scenario, we also only started
one person, who has the point 18 (yellow point)
as departure and the point 8 as target. As the
scenario is larger, we established a simulation
time of 30 seconds.
4 E XPERIMENT - PART 1 - R ESULTS
We executed the experiment 16 times, only
changing the decision solution of the au-
tonomous streetlights. In the first instance, we
set all outputs to zero (the zeroed solution) dur-
ing the whole simulation, which is supposed to
be the worst solution. For example, streetlights
never switch their lights ON. In the second in-
stance, we executed the experiment using the
best solution that was found by the learning
algorithm, according to the experiment presented
in [118]. Then, we executed the simulation for
9
the solution provided by each one of the 14
participants 3 .
To provide a controlled experiment and be
able to compare the different solutions, we
started with only one person in the scenario and
manually we set the parameters that were sup-
posed to be randomly selected, such as departure
and target points and broken lamps.
Each experiment execution consists of execut-
ing the simulated scenario three times: (i) night
(environmental light is equal to 0.0); (ii) late
afternoon (environmental light is equal to 0.5);
and (iii) morning (environmental light is equal
to 1.0). The main idea is to determine how the
solution behaves during different parts of the
day. Figure 10 depicts the percentage of energy
that was spent according to the environmental
light for each one of the 16 different solutions. As
we described previously, we also considered the
use of the wireless transmitter to calculate energy
consumption. As expected, as streetlights using
the zeroed decision never switch their lights ON
and never emit any signal, the energy consumed
using this solution is always zero. It is possible
to observe that only the solutions provided by
the learning algorithm and by the 5th and 11th
participants do not expend energy when the en-
vironmental light is maximum. In fact, according
to the proposed scenario, there is no reason to
turn ON streetlights during the period of the day
with maximum illumination.
100
90
80
.
70
0
60
50
(
40
30
.)1
20
10
0
.)1 ).5 .)1 ).5 % .)1 ).5
Fig. 10. Scenario1: Percentage of energy spent in different
parts of the day according to the participant solutions.
Figure 11 depicts the percentage of time that
was spent by the unique person in each one of
the simulations. As shown, the higher difference
between solutions occurs at night. If the time is
3. All files that were generated during the development
of this work, such as executable files and participants’
solutions results, are available at
http://www.inf.puc-rio.br/.̃nnascimento/projects.html
 10

100%, it means that the person did not complete simulation time did not allow the person to finish
the route, thus the solution did not work the route.

100 4.1 Discussion: Participants Knowledge in

90
80 IoT Versus Results
70
After executing the solution proposed by each
). ) (

60
50 participant, we connect that solution’s results
40
30
with the participant’s knowledge in the IoT do-
20 main, as shown in Table 3.
10
0
TABLE 3
1) 0 Correlation between participants expertises in the Internet
of Things with their solution results.
). ( ). ( % ). (

Experience
Fig. 11. Scenario1: Percentage of time spent by person with IoT Solution Does
to conclude his route based on different parts of the day Software Development Performance the
according to the participant solutions. Engineer (None/Low/ (Fitness solution
Medium/ Average) work?
Besides presenting the results of the different High)
1 High 55.48 Y
solutions in different parts of the day, the best
2 None 26.99 N
solution must be the one that presents the best 3 High 62.88 Y
result for the whole day. Thus, we calculated the 4 Low 62.49 Y
average of each one of the parameters (energy, 5 None 30.50 N
people, trip and fitness) that was achieved by 6 Low 51.09 Y
7 Medium 54.37 Y
solutions in different parts of the day. Figure 12 8 None 16.59 N
depicts a common average. We also calculated a 9 High 28.62 N
weighted average, taking into account the dura- 10 None 61.60 Y
tion of the parts of the day (we considered 12 11 None 29.67 N
12 Medium 47.81 Y
hours for the night period, 3h for dim and 9h for 13 None 30.32 N
the morning), but the results were very similar. 14 Low 56.91 Y
Learning 59.53 Y
100
zeroed 28.33 N
90
80
70 62.88
We observe a significant difference between
61.60
62.49 56.91 59.53
60 55.48
51.09
54.37
47.81 results from software engineers with any ex-
50
%

40
26.99
30
20
10
0
Fig. 12. Scenario1: Average of energy, trip and fitness
calculated for the different parts of the day according to the
participant solutions.
As shown in Figure 12, based on the fitness
average, three participants namely 3, 4 and 10
provided a solution slightly better than the so-
lution provided by the learning algorithm. Five
other participants provided a solution that works
and the remaining six provided a solution that
does not work. As explained earlier, we have
been considering an incorrect solution as one
in which the person did not finish the route
before the simulation ends. Even increasing the
29.67 30.32
perience in IoT development and results from
30.50 28.62 28.33
16.59 software engineers without experience in IoT
development. Participant 10 is the only individ-
ual without knowledge of IoT that provided a
solution that works and participant 9 is the only
Fitness individual with any knowledge of IoT that did
not provide a working solution.
4.2 Hypothesis Testing
In this section, we investigate the hypotheses
related to the solutions’ performance evaluation
(i.e H-RQ1 and H-RQ3), as presented in subsec-
tion 2.2. Thus, we performed statistical analyses,
as described by Peck and Devore [125], of the
measures presented in Table 3.
As shown in Table 4, we separated the results
of the experiments into two groups: i) software
engineers with IoT knowledge and ii) software
engineers without IoT knowledge. Then, we cal-
culated the mean and the standard deviation of

the results achieved by each group performing
the experiment and compare each result against
the value achieved using the ML-based solution.
4.2.1 How does the evaluation result from a ma-
chine learning-based solution differ from solutions
provided by IoT expert software engineers with
respect to their performance?
H - RQ1.
• H0. An ML-based approach does not im-
prove the performance of autonomous
things compared to solutions provided by
IoT expert software engineers.
• HA. An ML-based approach improves the
performance of autonomous things com-
pared to solutions provided by IoT expert
software engineers.
The first null hypothesis H0 claims that there
is no difference between the mean performance
for IoT expert software engineers’ solutions and
the ML-based approach solution. The alternative
hypothesis claims that the ML-based approach
solution improves the performance of the appli-
cation in comparison to IoT expert software en-
gineers’ solutions. Thus, the claim is that the true
IoT expert software engineers’ solutions mean
is below the performance achieved by the ML-
based approach, that is = 59.53.
Therefore, we used the ML performance as
our hypothesizedvalue to test the following one-
sided hypothesis:
H0: µs e = 59.53
H1: µs e < 59.53
where µs e denotes the true mean perfor-
mance for all IoT expert software engineers’ so-
lutions.
For instance, we restricted the population
sample to the number of software engineers that
confirmed having experience with developing
applications for the Internet of Things. As shown
in Table 4, the performance mean (x) of the IoT
expert software engineers’ solutions is 52.46 and
the standard deviation (σ ) is 10.91. To verify
if the data that we have is sufficient to accept
the alternative hypothesis, we need to verify the
probability of rejecting the null hypothesis [125].
Assuming that the H0 is true, and using a statis-
tical significance level [125] of 0.01 (the chance of
one in 100 of making an error), we computed the
test statistic (t − statistic), as follows [125]:
(x − hypothesizedvalue)
t − statistic : t = (5)
( √σn )
11
(52.46 − 59.53)
t − statistic : t = = −1.83 (6)
( 10.91
√ )
8
According to t-statistic theory, we can safely
reject our null hypothesis if the t − statistic
value is below the negative t − criticalvalue
(threshold) [125]. This negative t − criticalvalue
bounds the area of rejection of a T-distribution,
as shown in Figure 13. In our experiment, as we
specified a statistical significance level of 0.01,
the probability of getting a T-value less or equal
than the negative t − criticalvalue is 1%. We cal-
culated the tcriticalvalue of this T-distribution
according to the T-table presented in Peck and
Devore (2011, pg 791) [125]. Accordingly, for a
distribution with 7 degrees of freedom (see Table
4) and a confidence level of 99%, the negative
tcriticalvalue is -3.00. As we depicted in Figure
13, the test statistic of our sample is higher than
the tcriticalvalue.
Fig. 13. Hypothesis H - RQ1 Test Graph.
As the test statistic does not fall in the critical
region, we cannot safely reject this null hypoth-
esis. Based on a t-value of -1.83 and a degree of
freedom of 7, we could reject our null hypothesis
only if we had reduced the precision of our
experiment to 85%. Thus, we would fail to reject
the null hypothesis and would not accept the
alternative hypothesis. Therefore, we cannot
state that an ML-based approach improves the
performance of autonomous things compared
to solutions provided by IoT expert software
engineers.
4.2.2 How does the evaluation result from a ma-
chine learning-based solution differ from solutions
provided by software engineers without IoT skills
with respect to their performance?
H - RQ3.
• H0. An ML-based approach does not im-
prove the performance of autonomous
things compared to solutions provided by
 12

TABLE 4
Data to perform test statistic.

Degrees t
Standard
n Highest Mean of critical
Variable Median deviation
samples value x freedom value
σ
(n-1) (.99%)
Software
14 62.88 43.95 49.45 16.00 13 2.65
Engineers
Software
Engineers
8 62.88 52.46 54.92 10.91 7 3.00
with IoT
knowledge
Software
Engineers
6 61.60 32.61 30.00 15.15 5 3.37
without IoT
knowledge
Machine-
learning
1 59.53
based
approach

software engineers without experience in As the t − statistic value is below the

IoT development.
• HA. An ML-based approach improves the
performance of autonomous things com-
pared to solutions provided by software
engineers without experience in IoT de-
velopment.
For instance, we restricted the population
sample to the number of software engineers that
confirmed not having experience with develop-
ing applications for the Internet of Things. As
shown in Table 4, the performance mean (x) of
the solutions from software engineers without
experience in IoT development is 32.61 and the
standard deviation (σ ) is 15.15. Thus, the
(32.61 − 59.53)
t − statistic : t = = −4.35 (7)
( 15.15
√ )
6 for the zeroed solution.
As shown in Table 4, this T-distribution has 5
degrees of freedom. Thus, for a confidence level
of 99%, the negative tcriticalvalue is -3.37. As
we depicted in Figure 14, the test statistic of our
sample is below the tcriticalvalue .
Fig. 14. Hypothesis H - RQ3 Test Graph.
negative tcriticalvalue (-4.35 < −3.37), we can
safely reject the null hypothesis, assuring that
the error chancing of making an error is lower
than 1%. Therefore, we accepted the alternative
hypothesis: An ML-based approach improves the
performance of autonomous things compared to
solutions provided by software engineers with no
experience in IoT development.
5 E XPERIMENT - PART 2 - R ESULTS
As explained previously, the second part of the
experiment consists of translating the solution
provided by machine learning and participants
to an unknown environment. In this second part
of the experiment, we also executed the simula-
tion 16 times: for each one of the participants’
solutions, for the machine-learning solution and
Table 5 shows the results that were achieved
by the different solutions at night in a simulation
of 30 seconds. As shown, most of the solutions
did not work. The person in these simulations
did not finish the route even when we increased
the simulation time. Only the solution provided
by the machine-learning algorithm and by par-
ticipant 12 worked. Remember, this scenario was
not used by the machine-learning algorithm dur-
ing the training process. This solution was pro-
vided through machine learning for the first sce-
nario and it was just reused in this new scenario.
In other words, we did not restart the machine-
learning process.
We selected only those solutions that worked
and verified their results for the other periods of
 13

the day (morning and late afternoon). As shown of 14 engineers, only one participant, who has
in Table 6, when considering the whole day, the experience with IoT development, provided a
machine-learning approach presented the best solution that worked.
result. Because the average time for the trip
was a little higher using the machine-learning
approach, the difference in energy consumption
Bad
between the two solutions is considerably higher.
93% 0% Work better than ML
7%
TABLE 5 Work
Using the same solution in a different environment - only
at night.

Software Fig. 15. Participants’ solution results in the second sce-

Energy% People% Trip% Fitness
Engineer nario.
1 6.50 0 100 -42.60
2 2.77 0 100 -41.11 Therefore, we can safely reject the null hy-
3 6.62 0 100 -42.65
4 4.30 0 100 -41.72 pothesis and accept both alternative hypotheses:
5 2.58 0 100 -41.03
6 6.88 0 100 -42.75 1) H-RQ2: H1: An ML-based approach in-
7 8.33 0 100 -43.33 creases the reuse of autonomous things
8 2.33 0 100 -60.26 compared to solutions provided by IoT
9 3.77 0 100 -41.51 expert software engineers.
10 3.78 0 100 -60.18
11 11.36 0 100 -44.54 2) H-RQ4: H1: 2) An ML-based approach
12 50.56 100 42.22 54.43 increases the reuse of autonomous things
13 2.77 0 100 -41.11 compared to solutions provided by soft-
14 4.50 0 100 -41.80 ware engineers with no experience with
Learning 24.44 100 61.11 53.55
zeroed 0 0 100 -40
IoT development.

TABLE 6 6 D ISCUSSION
Using the same solution in a different environment - day
average.
In this section, we analyze the empirical exper-
imental results to understand which tasks are
Energy% People% Trip% Fitness better performed by humans and which by al-
Average gorithms. This is important for selecting whether
Participant 50.52 100 38.14 56.90 software engineers or machine learning can ac-
12 complish a specific task better.
Average
8.46 100 46.29 68.83 In our empirical study, in which we have as-
Learning
sessed performance and reuse tasks, we accepted
three alternative hypotheses and rejected one:
5.1 Hypothesis Testing
In this section, we investigate the hypotheses re- Accepted:
lated to the solutions’ reuse evaluation, that is H- 1) An ML-based approach improves the
RQ2 and H-RQ4, as presented in subsection 2.2. performance of autonomous things com-
Their alternative hypotheses state that an ML- pared to solutions provided by software
based approach improves the performance of au- engineers without experience with IoT
tonomous things compared to solutions provided development.
by software engineers, software engineers with 2) An ML-based approach increases the
experience in IoT development, and software en- reuse of autonomous things compared
gineers without experience in IoT development, to solutions provided by IoT expert soft-
respectively. We planned to perform a statisti- ware engineers.
cal development to evaluate these hypotheses. 3) An ML-based approach increases the
However, as depicted in Figure 15, in the new reuse of autonomous things compared
scenario, 0% of participants provided a result to solutions provided by software engi-
better than the result provided by the machine- neers without experience with IoT devel-
learning solution. In addition, from the group opment.

Rejected:
1) An ML-based approach improves the
performance of autonomous things com-
pared to solutions provided by IoT ex-
pert software engineers.
Based on these results, we have found ev-
idence that the use of machine-learning tech-
niques can perform some SE tasks better than
software engineers, considering solutions that
improve performance and increase reuse. As il-
lustrated in the experimental results, only one
of the 14 software engineers provided a solution
that could be reused in a new scenario. Further
none of those software engineers provided a so-
lution that works better than the ML’s solution
in this new scenario. If the flexibility of the ap-
plication is the most important factor, based on
our results, we can safely recommend the use of
machine learning.
However, if we had considered performance
as the only important factor to evaluate the qual-
ity of these solutions, we have found evidence
that software engineers can perform SE tasks bet-
ter than machine learning, considering “better”
as a solution that improves performance. As de-
scribed in our experiments, we cannot state that
ML improves the performance of an application
in comparison to solutions provided by domain
expert software engineers. This is also an interest-
ing result as many researchers, especially in the
IoT domain, have strictly focused on automating
software development.
In brief, our experiment indicates that in some
cases, software engineers outperform machine-
learning algorithms, whereas in other cases, they
do not. The evidence shows that it is important to
know which one performs better in different sit-
uations in order to determine ways for software
engineers to work cooperatively and effectively
with automated machine-learning procedures.
7 T HREATS TO VALIDITY
Although we have designed and conducted the
experiments carefully, there are always factors
that can challenge the experiments validity. Some
threats to validity as described in [123] could
indeed limit the legitimacy of our results. In this
section, we present the actions taken to mitigate
their impact of these factors on the research re-
sults.
As Oizumi et al. (2017) report in [126], the
number of participants in the study can be a
threat to validity. In addition, Fernandes et al
14
(2016) [127] report the diversity of participants as
another possible threat. Therefore, in our study,
we needed to be aware of at least two threats
to validity namely: we have selected a sample of
only 14 participants, which may not be enough
to achieve conclusive results; and our sample
consisted of only graduate students from two
Brazilian universities. Such a group may not be
representative of all software engineers, who may
have substantially more professional experience
and background.
To mitigate the problems of the number of
participants and their diversity, we selected our
participants carefully. All of them have at least
two years of experience with software develop-
ment. In addition, we allowed participants to
solve the problem by manipulating a pseudocode
version, thereby avoiding gaps in the partici-
pants’ knowledge, such as experience with a par-
ticular programming language or architecture.
Note that a survey was used to select participants
and they all indicated a level of experience with
pseudocode. The pseudocode provided by each
participant was carefully translated into Java as
this is the language supported by the Framework
for the Internet of Things.
Oizumi et al. (2017) reported a third threat
to validity in [126], namely, possible misunder-
standings during the study. To mitigate this prob-
lem of misunderstandings, we asked all partici-
pants to write about their understanding of the
problem both before and after applying the solu-
tion. All participants indicated that they under-
stood the task completely. We also asked them
about their confidence in their proposed solu-
tion. Most of them evaluated their own solution
with the highest grade, allowing us to increase
our confidence in the experimental results. In
addition, we assisted the participants during the
entire study, making sure they understood the
experimental task
8 R ELATED W ORK
Comparing intelligent machines to the ability of
a person to solve a particular problem is not
a new approach. This kind of discussion has
been promoted since the beginning of Artificial
Intelligence. For example, in 1997, an important
moment in the history of technology happened
with Garry Kasparov’s 1997 chess match against
the IBM supercomputer Deep Blue [128].
Recently, Silver et al. (2016, 2017) [129], [130]
published a paper in the Nature Science Journal,
comparing the performance of a ML technique

against the results achieved by the world cham-
pion in the game of Go. In [130], Silver et al.
(2017) state that their program “achieved super-
human performance.”
Whiteson et al. [122] indirectly performed this
comparison, by evaluating the use of three dif-
ferent approaches of the neuroevolution learning
algorithm to solve the same tasks: (i) coevolution,
that is mostly unassisted by human knowledge;
(ii) layered learning, that is highly assisted; and
(iii) concurrent layered learning, that is a mixed
approach. The authors state that their results
“demonstrate that the appropriate level of hu-
man assistance depends critically on the diffi-
culty of the problem.”
Furthermore, there is also a new approach
in machine learning, called Automatic Machine
Learning (Auto-ML) [100], which uses learning
to set the parameters of a learning algorithm au-
tomatically. In a traditional approach, a software
engineer with machine learning skills is respon-
sible for finding a good configuration for the
algorithm parameters. Zoth and Lee [100] present
an Auto-ML-based approach to design a neural
network to classify images of a specific dataset.
In addition, they compared their results with
the previous state-of-the-art model, which was
designed by an ML expert engineer. According
to Zoth and Lee [100] , their AutoML-based ap-
proach “can design a novel network architecture
that rivals the best human-invented architecture
in terms of test set accuracy.” Zoth and Lee
also showed that a machine-learning technique
is capable of beating a software engineer with
ML skills in a specific software engineering task,
but the authors do not discuss this subject in the
paper.
Our paper appears to be the first to pro-
vide an empirical study to investigate the use
of a machine-learning techniques to solve a
problem in the field of Software Engineering,
by comparing the solution provided by a ML-
based approach against solutions provided by
software engineers.
9 C ONCLUSION AND F UTURE W ORK
Several researchers have proposed the use of
machine-learning techniques to automate soft-
ware engineering tasks. However, most of these
approaches do not direct efforts toward asking
whether ML-based procedures have higher suc-
cess rates than current standard and manual
practices. A relevant question in this potential
15
line of investigation is: “Could a software engi-
neer solve a specific development task better than
an ML algorithm?”. Indeed, it is fundamental to
evaluate which tasks are better performed by en-
gineers or ML procedures so that they can work
together more effectively and also provide more
insight into novel human-in-the-loop machine-
learning approaches to support SE tasks.
This paper appears to be the first to pro-
vide an empirical study comparing how soft-
ware engineers and machine-learning algorithms
achieve performance and reuse tasks. In brief,
as a result of our experiment, we have found
evidence that in some cases, software engineers
outperform machine-learning algorithms, and in
other cases, they do not. Further, as is typical in
experimental studies, although we have designed
and conducted the experiment carefully, there are
always factors that can threaten the experiment’s
validity. For example, some threats include the
number and diversity of the software engineers
involved in our experiment.
Understanding how software engineers fare
against ML algorithms is essential to support
new methodologies for developing human-in-
the-loop approaches in which machine learning
automated procedures assist software developers
in achieving their tasks. For example, method-
ologies to define which agent (engineers or auto-
mated ML procedure) should execute a specific
task in a software development set. Based on
this understanding, these methodologies can pro-
vide a basis for software engineers and machine
learning algorithms to cooperate in Software En-
gineering development more effectively.
Future work to extend the proposed experi-
ment includes: (i) conducting further empirical
studies to assess other SE tasks, such as design,
maintenance and testing; (ii) experimenting with
other machine-learning algorithms such as re-
inforcement learning and backpropagation; and
(iii) using different criteria to evaluate task exe-
cution.
Possible tasks that could be investigated (refer
to (i)) include programming tasks, in which case
tasks performed by software development teams
and ML algorithms are compared. For example,
we could invite software developers from the
team with the highest score in the last ACM Inter-
national Collegiate Programming Contest [131],
which is one of the most important programming
championships in the world, to be involved in
this comparison. This competition evaluates the
capability of software engineers to solve complex

software problems. Software engineers are classi-
fied according to the number of right solutions,
performance of the solutions and development
time.
Another line of investigation could address
the use of different qualitative or quantitative
methodologies. For example, the task execu-
tion comparison could rely on reference per-
formances, such as the performance of highly
successful performers [100], [129], [130] . This
research work can also be extended by propos-
ing, based on the comparison between the per-
formance of engineers and ML algorithms, a
methodology for more effective task allocation.
This methodology could, in principle, lead to
more effective ways to allocate tasks such as soft-
ware development in cooperative work involv-
ing humans and automated procedures. Such
human-in-the-loop approaches, which take into
account the strengths and weaknesses of humans
and machine learning algorithms, are fundamen-
tal to provide a basis for cooperative work in
software engineering and possibly in other areas.
ACKNOWLEDGMENTS
This work has been supported by the Labora-
tory of Software Engineering (LES) at PUC-Rio.
Our thanks to CAPES, CNPq, FAPERJ and PUC-
Rio for their support through scholarships and
fellowships. We would also like to thank the
software engineers who participated in our ex-
periment.
R EFERENCES
[1] F. Brooks and H. Kugler, No silver bullet.
April, 1987.
[2] R. S. Pressman, Software engineering: a prac-
titioner’s approach. Palgrave Macmillan,
2005.
[3] Q. Zhang, “Software developments,” Engi-
neering Automation for Reliable Software, p.
292, 2000.
[4] J. O. Kephart, “Research challenges of auto-
nomic computing,” in Software Engineering,
2005. ICSE 2005. Proceedings. 27th Interna-
tional Conference on. IEEE, 2005, pp. 15–22.
[5] J. Mostow, “Foreword what is ai? and what
does it have to do with software engineer-
ing?” IEEE Transactions on Software Engi-
neering, vol. 11, no. 11, p. 1253, 1985.
[6] D. Barstow, “Artificial intelligence and soft-
ware engineering,” in Proceedings of the 9th
16
international conference on Software Engineer-
ing. IEEE Computer Society Press, 1987,
pp. 200–211.
[7] D. Partridge, “Artificial intelligence and
software engineering: a survey of possibil-
ities,” Information and Software Technology,
vol. 30, no. 3, pp. 146–152, 1988.
[8] L. C. Cheung, S. Ip, and T. Holden, “Survey
of artificial intelligence impacts on infor-
mation systems engineering,” Information
and Software Technology, vol. 33, no. 7, pp.
499–508, 1991.
[9] D. Partridge, Artificial Intelligence in Soft-
ware Engineering. Wiley Online Library,
1998.
[10] A. Van Lamsweerde and L. Willemet, “In-
ferring declarative requirements specifica-
tions from operational scenarios,” IEEE
Transactions on Software Engineering, vol. 24,
no. 12, pp. 1089–1114, 1998.
[11] G. D. Boetticher, “Using machine learn-
ing to predict project effort: Empirical case
studies in data-starved domains,” in Model
Based Requirements Workshop. Citeseer,
2001, pp. 17–24.
[12] F. Padberg, T. Ragg, and R. Schoknecht,
“Using machine learning for estimating the
defect content after an inspection,” IEEE
Transactions on Software Engineering, vol. 30,
no. 1, pp. 17–28, 2004.
[13] D. Zhang, “Applying machine learning al-
gorithms in software development,” in The
Proceedings of 2000 Monterey Workshop on
Modeling Software System Structures, 2000,
pp. 275–285.
[14] ——, “Machine learning in value-based
software test data generation,” in Tools with
Artificial Intelligence, 2006. ICTAI’06. 18th
IEEE International Conference on. IEEE,
2006, pp. 732–736.
[15] D. Zhang and J. J. Tsai, Machine learning
applications in software engineering. World
Scientific, 2005, vol. 16.
[16] D. Zhang, “Machine learning and value-
based software engineering: a research
agenda.” in SEKE, 2008, pp. 285–290.
[17] T. M. Khoshgoftaar, “Introduction to the
special issue on quality engineering with
computational intelligence,” 2003.
[18] D. Zhang, “Machine learning and value-
based software engineering,” in Software
Applications: Concepts, Methodologies, Tools,
and Applications. IGI Global, 2009, pp.
3325–3339.

[19] D. Zhang and J. J. Tsai, “Machine learning
and software engineering,” in Tools with
Artificial Intelligence, 2002.(ICTAI 2002). Pro-
ceedings. 14th IEEE International Conference
on. IEEE, 2002, pp. 22–29.
[20] M. D. Kramer and D. Zhang, “Gaps: a
genetic programming system,” in Computer
Software and Applications Conference, 2000.
COMPSAC 2000. The 24th Annual Interna-
tional. IEEE, 2000, pp. 614–619.
[21] A. Holzinger, M. Plass, K. Holzinger, G. C.
Crişan, C.-M. Pintea, and V. Palade, “To-
wards interactive machine learning (iml):
applying ant colony algorithms to solve
the traveling salesman problem with the
human-in-the-loop approach,” in Interna-
tional Conference on Availability, Reliability,
and Security. Springer, 2016, pp. 81–95.
[22] A. Holzinger, “Interactive machine learn-
ing for health informatics: when do we
need the human-in-the-loop?” Brain Infor-
matics, vol. 3, no. 2, pp. 119–131, 2016.
[23] S. Easterbrook, J. Singer, M.-A. Storey, and
D. Damian, “Selecting empirical methods
for software engineering research,” Guide to
advanced empirical software engineering, pp.
285–311, 2008.
[24] H. A. Simon, “Whether software engineer-
ing needs to be artificially intelligent,” IEEE
Transactions on Software Engineering, no. 7,
pp. 726–732, 1986.
[25] I. Sommerville, “Artificial intelligence and
systems engineering,” Prospects for Arti-
ficial Intelligence: Proceedings of AISB’93,
29 March-2 April 1993, Birmingham, UK,
vol. 17, p. 48, 1993.
[26] R. S. Michalski, J. G. Carbonell, and T. M.
Mitchell, Machine learning: An artificial in-
telligence approach. Springer Science &
Business Media, 2013.
[27] A. Marchetto and A. Trentini, “Evaluating
web applications testability by combining
metrics and analogies,” in Information and
Communications Technology, 2005. Enabling
Technologies for the New Knowledge Society:
ITI 3rd International Conference on. IEEE,
2005, pp. 751–779.
[28] S. Bouktif, F. Ahmed, I. Khalil, and G. Anto-
niol, “A novel composite model approach
to improve ality prediction,” Information
and Software Technology, vol. 52, no. 12, pp.
1298–1311, 2010.
[29] L. Radlinski, “A survey of bayesian net
models for software development effort
17
prediction,” International Journal of Software
Engineering and Computing, vol. 2, no. 2, pp.
95–109, 2010.
[30] W. Zhang, Y. Yang, and Q. Wang, “Han-
dling missing data in software effort pre-
diction with naive bayes and em algo-
rithm,” in Proceedings of the 7th International
Conference on Predictive Models in Software
Engineering. ACM, 2011, p. 4.
[31] Ł. Radliński, “A framework for inte-
grated software quality prediction using
bayesian nets,” Computational Science and
Its Applications-ICCSA 2011, pp. 310–325,
2011.
[32] P. O. O. Sack, M. Bouneffa, Y. Maweed,
and H. Basson, “On building an integrated
and generic platform for software quality
evaluation,” in Information and Communica-
tion Technologies, 2006. ICTTA’06. 2nd, vol. 2.
IEEE, 2006, pp. 2872–2877.
[33] M. Reformat and D. Zhang, “Introduc-
tion to the special issue on:“software
quality improvements and estimations
with intelligence-based methods”,” Soft-
ware Quality Journal, vol. 15, no. 3, pp. 237–
240, 2007.
[34] B. Twala, M. Cartwright, and M. Shepperd,
“Applying rule induction in software pre-
diction,” in Advances in Machine Learning
Applications in Software Engineering. IGI
Global, 2007, pp. 265–286.
[35] V. U. Challagulla, F. B. Bastani, and I.-L.
Yen, “High-confidence compositional relia-
bility assessment of soa-based systems us-
ing machine learning techniques,” in Ma-
chine Learning in Cyber Trust. Springer,
2009, pp. 279–322.
[36] R. C. Veras, S. R. Meira, A. L. Oliveira,
and B. J. Melo, “Comparative study of clus-
tering techniques for the organization of
software repositories,” in Hybrid Intelligent
Systems, 2007. HIS 2007. 7th International
Conference on. IEEE, 2007, pp. 372–377.
[37] I. Birzniece and M. Kirikova, “Interactive
inductive learning service for indirect anal-
ysis of study subject compatibility,” in Pro-
ceedings of the BeneLearn, 2010, pp. 1–6.
[38] D. B. Hanchate, “Analysis, mathemati-
cal modeling and algorithm for software
project scheduling using bcga,” in In-
telligent Computing and Intelligent Systems
(ICIS), 2010 IEEE International Conference on,
vol. 3. IEEE, 2010, pp. 1–7.
[39] Z. Xu and B. Song, “A machine learning ap-

plication for human resource data mining
problem,” Advances in Knowledge Discovery
and Data Mining, pp. 847–856, 2006.
[40] J. Wen, S. Li, Z. Lin, Y. Hu, and C. Huang,
“Systematic literature review of machine
learning based software development ef-
fort estimation models,” Information and
Software Technology, vol. 54, no. 1, pp. 41–
59, 2012.
[41] E. Rashid, S. Patnayak, and V. Bhattacher-
jee, “A survey in the area of machine learn-
ing and its application for software quality
prediction,” ACM SIGSOFT Software Engi-
neering Notes, vol. 37, no. 5, pp. 1–7, 2012.
[42] H. A. Al-Jamimi and M. Ahmed, “Machine
learning-based software quality prediction
models: state of the art,” in Information
Science and Applications (ICISA), 2013 Inter-
national Conference on. IEEE, 2013, pp. 1–4.
[43] Ł. Radliński, “Enhancing bayesian network
model for integrated software quality pre-
diction,” in Proc. Fourth International Confer-
ence on Information, Process, and Knowledge
Management, Valencia. Citeseer, 2012, pp.
144–149.
[44] F. Pinel, P. Bouvry, B. Dorronsoro, and S. U.
Khan, “Savant: Automatic parallelization
of a scheduling heuristic with machine
learning,” in Nature and Biologically Inspired
Computing (NaBIC), 2013 World Congress on.
IEEE, 2013, pp. 52–57.
[45] D. Novitasari, I. Cholissodin, and W. F.
Mahmudy, “Optimizing svr using local
best pso for software effort estimation,”
Journal of Information Technology and Com-
puter Science, vol. 1, no. 1, 2016.
[46] Ł. Radliński, “Towards expert-based mod-
elling of integrated software quality,” Jour-
nal of Theoretical and Applied Computer Sci-
ence, vol. 6, no. 2, pp. 13–26, 2012.
[47] T. Rongfa, “Defect classification method
for software management quality control
based on decision tree learning,” in Ad-
vanced Technology in Teaching-Proceedings
of the 2009 3rd International Conference on
Teaching and Computational Science (WTCS
2009). Springer, 2012, pp. 721–728.
[48] R. Rana and M. Staron, “Machine learn-
ing approach for quality assessment and
prediction in large software organizations,”
in Software Engineering and Service Science
(ICSESS), 2015 6th IEEE International Con-
ference on. IEEE, 2015, pp. 1098–1101.
[49] H. Wang, M. Kessentini, W. Grosky, and
18
H. Meddeb, “On the use of time series
and search based software engineering for
refactoring recommendation,” in Proceed-
ings of the 7th International Conference on
Management of computational and collective
intElligence in Digital EcoSystems. ACM,
2015, pp. 35–42.
[50] V. U. Challagulla, F. B. Bastani, I.-L.
Yen, and R. A. Paul, “Empirical assess-
ment of machine learning based soft-
ware defect prediction techniques,” in
Object-Oriented Real-Time Dependable Sys-
tems, 2005. WORDS 2005. 10th IEEE Inter-
national Workshop on. IEEE, 2005, pp. 263–
270.
[51] K. Kaminsky and G. Boetticher, “Build-
ing a genetically engineerable evolvable
program (geep) using breadth-based ex-
plicit knowledge for predicting software
defects,” in Fuzzy Information, 2004. Process-
ing NAFIPS’04. IEEE Annual Meeting of the,
vol. 1. IEEE, 2004, pp. 10–15.
[52] ——, “How to predict more with less, de-
fect prediction using machine learners in
an implicitly data starved domain,” in The
8th world multiconference on systemics, cyber-
netics and informatics, Orlando, FL. Citeseer,
2004.
[53] K. Kaminsky and G. D. Boetticher, “Better
software defect prediction using equalized
learning with machine learners,” Knowledge
Sharing and Collaborative Engineering, 2004.
[54] O. Kutlubay and A. Bener, “A machine
learning based model for software defect
prediction,” working paer, Boaziçi University,
Computer Engineering Department, 2005.
[55] X. Ren, “Learn to predict “affecting
changes” in software engineering,” 2003.
[56] E. Ceylan, F. O. Kutlubay, and A. B. Bener,
“Software defect identification using ma-
chine learning techniques,” in Software En-
gineering and Advanced Applications, 2006.
SEAA’06. 32nd EUROMICRO Conference on.
IEEE, 2006, pp. 240–247.
[57] Y. Kastro and A. B. Bener, “A defect pre-
diction method for software versioning,”
Software Quality Journal, vol. 16, no. 4, pp.
543–562, 2008.
[58] O. Kutlubay, B. Turhan, and A. B. Bener, “A
two-step model for defect density estima-
tion,” in Software Engineering and Advanced
Applications, 2007. 33rd EUROMICRO Con-
ference on. IEEE, 2007, pp. 322–332.
[59] A. S. Namin and M. Sridharan, “Bayesian

reasoning for software testing,” in Proceed-
ings of the FSE/SDP workshop on Future of
software engineering research. ACM, 2010,
pp. 349–354.
[60] C. Murphy and G. Kaiser, “Metamor-
phic runtime checking of non-testable pro-
grams,” Columbia University Dept of Com-
puter Science Tech Report cucs-042-09, p.
9293, 2009.
[61] W. Afzal, R. Torkar, R. Feldt, and
T. Gorschek, “Genetic programming for
cross-release fault count predictions in
large and complex software projects,” Evo-
lutionary Computation and Optimization Al-
gorithms in Software Engineering, pp. 94–126,
2010.
[62] C. Murphy et al., “Using metamorphic test-
ing at runtime to detect defects in applica-
tions without test oracles,” 2008.
[63] D. Qiu, S. Fang, and Y. Li, “A framework to
discover potential deviation between pro-
gram and requirement through mining ob-
ject graph,” in Computer Application and Sys-
tem Modeling (ICCASM), 2010 International
Conference on, vol. 4. IEEE, 2010, pp. V4–
110.
[64] C. Murphy, G. E. Kaiser et al., “Automatic
detection of defects in applications without
test oracles,” Dept. Comput. Sci., Columbia
Univ., New York, NY, USA, Tech. Rep. CUCS-
027-10, 2010.
[65] W. Afzal, “Search-based approaches to soft-
ware fault prediction and software test-
ing,” Ph.D. dissertation, Blekinge Institute
of Technology, 2009.
[66] M. K. Taghi, B. Cukic, and N. Seliya, “An
empirical assessment on program module-
order models,” Quality Technology & Quan-
titative Management, vol. 4, no. 2, pp. 171–
190, 2007.
[67] J. H. Wang, N. Bouguila, and T. Bdiri, “Em-
pirical evaluation of selected algorithms for
complexity-based classification of software
modules and a new model,” in Intelligent
Systems: From Theory to Practice. Springer,
2010, pp. 99–131.
[68] H. Jin, Y. Wang, N.-W. Chen, Z.-J. Gou,
and S. Wang, “Artificial neural network for
automatic test oracles generation,” in Com-
puter Science and Software Engineering, 2008
International Conference on, vol. 2. IEEE,
2008, pp. 727–730.
[69] J. Ferzund, S. N. Ahsan, and F. Wotawa,
“Automated classification of faults in pro-
19
gramms using machine learning tech-
niques,” in Artificial Intelligence Techniques
in Software Engineering Workshop, 2008.
[70] O. Maqbool and H. Babri, “Bayesian learn-
ing for software architecture recovery,” in
Electrical Engineering, 2007. ICEE’07. Inter-
national Conference on. IEEE, 2007, pp. 1–6.
[71] A. Okutan, “Software defect prediction us-
ing bayesian networks and kernel meth-
ods,” Ph.D. dissertation, ISIK UNIVER-
SITY, 2012.
[72] D. Cotroneo, R. Pietrantuono, and S. Russo,
“A learning-based method for combining
testing techniques,” in Proceedings of the
2013 International Conference on Software En-
gineering. IEEE Press, 2013, pp. 142–151.
[73] D. Zhang, “A value-based framework for
software evolutionary testing,” in Advances
in Abstract Intelligence and Soft Computing.
IGI Global, 2013, pp. 355–373.
[74] A. Okutan and O. T. Yıldız, “Software de-
fect prediction using bayesian networks,”
Empirical Software Engineering, vol. 19, no. 1,
pp. 154–181, 2014.
[75] S. Agarwal and D. Tomar, “A feature se-
lection based model for software defect
prediction,” assessment, vol. 65, 2014.
[76] G. Abaei and A. Selamat, “Important is-
sues in software fault prediction: A road
map,” in Handbook of Research on Emerging
Advancements and Technologies in Software
Engineering. IGI Global, 2014, pp. 510–539.
[77] A. Okutan and O. T. Yildiz, “A novel kernel
to predict software defectiveness,” Journal
of Systems and Software, vol. 119, pp. 109–
121, 2016.
[78] X.-d. Mu, R.-h. Chang, and L. Zhang,
“Software defect prediction based on com-
petitive organization coevolutionary algo-
rithm,” Journal of Convergence Information
Technology (JCIT) Volume7, Number5, 2012.
[79] J. Cahill, J. M. Hogan, and R. Thomas,
“Predicting fault-prone software modules
with rank sum classification,” in Software
Engineering Conference (ASWEC), 2013 22nd
Australian. IEEE, 2013, pp. 211–219.
[80] R. Rana, M. Staron, C. Berger, J. Hans-
son, M. Nilsson, and W. Meding, “The
adoption of machine learning techniques
for software defect prediction: An initial
industrial validation,” in Joint Conference
on Knowledge-Based Software Engineering.
Springer, 2014, pp. 270–285.
[81] T. Schulz, Ł. Radliński, T. Gorges, and

W. Rosenstiel, “Predicting the flow of de-
fect correction effort using a bayesian net-
work model,” Empirical Software Engineer-
ing, vol. 18, no. 3, pp. 435–477, 2013.
[82] E. Rashid, “R4 model for case-based rea-
soning and its application for software
fault prediction,” International Journal of
Software Science and Computational Intelli-
gence (IJSSCI), vol. 8, no. 3, pp. 19–38, 2016.
[83] ——, “Improvisation of case-based reason-
ing and its application for software fault
prediction,” International Journal of Services
Technology and Management, vol. 21, no. 4-6,
pp. 214–227, 2015.
[84] J. K. Chhabra and A. Parashar, “Predic-
tion of changeability for object oriented
classes and packages by mining change
history,” in Electrical and Computer Engi-
neering (CCECE), 2014 IEEE 27th Canadian
Conference on. IEEE, 2014, pp. 1–6.
[85] G. Spanoudakis, A. S. d. Garcez, and
A. Zisman, “Revising rules to capture re-
quirements traceability relations: A ma-
chine learning approach.” in SEKE, 2003,
pp. 570–577.
[86] M. Shin and A. Goel, “Modeling soft-
ware component criticality using a ma-
chine learning approach,” Artificial Intelli-
gence and Simulation, pp. 440–448, 2005.
[87] J. S. Shirabad, “Predictive techniques in
software engineering,” in Encyclopedia of
Machine Learning. Springer, 2011, pp. 782–
789.
[88] A. A. Araújo, M. Paixao, I. Yeltsin, A. Dan-
tas, and J. Souza, “An architecture based
on interactive optimization and machine
learning applied to the next release prob-
lem,” Automated Software Engineering, pp.
1–49, 2016.
[89] T. Tourwé, J. Brichau, A. Kellens, and
K. Gybels, “Induced intentional software
views,” Computer Languages, Systems &
Structures, vol. 30, no. 1, pp. 35–47, 2004.
[90] J. S. Di Stefano and T. Menzies, “Machine
learning for software engineering: Case
studies in software reuse,” in Tools with
Artificial Intelligence, 2002.(ICTAI 2002). Pro-
ceedings. 14th IEEE International Conference
on. IEEE, 2002, pp. 246–251.
[91] J. Fu, F. B. Bastani, and I.-L. Yen, “Auto-
mated ai planning and code pattern based
code synthesis,” in Tools with Artificial In-
telligence, 2006. ICTAI’06. 18th IEEE Interna-
tional Conference on. IEEE, 2006, pp. 540–
20
546.
[92] J. Fu, F. B. Bastani, I.-L. YEN et al.,
“Semantic-driven component-based auto-
mated code synthesis,” Semantic Comput-
ing, pp. 249–283, 2010.
[93] A. Katasonov, O. Kaykova, O. Khriyenko,
S. Nikitin, and V. Y. Terziyan, “Smart
semantic middleware for the internet of
things.” ICINCO-ICSO, vol. 8, pp. 169–178,
2008.
[94] L. Baresi, S. Guinea, and A. Shahzada,
“Short paper: Harmonizing heterogeneous
components in sesame,” in Internet of
Things (WF-IoT), 2014 IEEE World Forum on.
IEEE, 2014, pp. 197–198.
[95] L. Zhu, H. Cai, and L. Jiang, “Minson: A
business process self-adaptive framework
for smart office based on multi-agent,” in
e-Business Engineering (ICEBE), 2014 IEEE
11th International Conference on. IEEE, 2014,
pp. 31–37.
[96] J. F. De Paz, J. Bajo, S. Rodrı́guez, G. Vil-
larrubia, and J. M. Corchado, “Intelligent
system for lighting control in smart cities,”
Information Sciences, vol. 372, pp. 241–255,
2016.
[97] I. Birzniece, “The use of inductive learning
in information systems,” in Proceedings of
the 16th International Conference on Informa-
tion and Software Technologies, 2010, pp. 95–
101.
[98] D. Alrajeh, A. Russo, and S. Uchitel, “Infer-
ring operational requirements from scenar-
ios and goal models using inductive learn-
ing,” in Proceedings of the 2006 international
workshop on Scenarios and state machines:
models, algorithms, and tools. ACM, 2006,
pp. 29–36.
[99] A. M. Sharifloo, A. Metzger, C. Quinton,
L. Baresi, and K. Pohl, “Learning and evo-
lution in dynamic software product lines,”
in Proceedings of the 11th International Sym-
posium on Software Engineering for Adaptive
and Self-Managing Systems. ACM, 2016, pp.
158–164.
[100] B. Zoph and Q. V. Le, “Neural architecture
search with reinforcement learning,” arXiv
preprint arXiv:1611.01578, 2016.
[101] N. M. do Nascimento and C. J. P. de Lu-
cena, “Fiot: An agent-based framework for
self-adaptive and self-organizing applica-
tions based on the internet of things,” Infor-
mation Sciences, vol. 378, pp. 161–176, 2017.
[102] F. Jacob and R. Tairas, “Code template in-

ference using language models,” in Proceed-
ings of the 48th Annual Southeast Regional
Conference. ACM, 2010, p. 104.
[103] B. Amal, M. Kessentini, S. Bechikh, J. Dea,
and L. B. Said, “On the use of machine
learning and search-based software engi-
neering for ill-defined fitness function: a
case study on software refactoring,” in In-
ternational Symposium on Search Based Soft-
ware Engineering. Springer, 2014, pp. 31–
45.
[104] Y. Peng, G. Wang, and H. Wang, “User
preferences based software defect detection
algorithms selection using mcdm,” Informa-
tion Sciences, vol. 191, pp. 3–13, 2012.
[105] H. A. Abbass, E. Petraki, K. Merrick, J. Har-
vey, and M. Barlow, “Trusted autonomy
and cognitive cyber symbiosis: Open chal-
lenges,” Cognitive computation, vol. 8, no. 3,
pp. 385–408, 2016.
[106] W. G. Baxt, “Use of an artificial neural
network for the diagnosis of myocardial
infarction,” Annals of internal medicine, vol.
115, no. 11, pp. 843–848, 1991.
[107] M. A. Mazurowski, P. A. Habas, J. M.
Zurada, J. Y. Lo, J. A. Baker, and G. D.
Tourassi, “Training neural network clas-
sifiers for medical decision making: The
effects of imbalanced datasets on classifica-
tion performance,” Neural networks, vol. 21,
no. 2, pp. 427–436, 2008.
[108] N. M. do Nascimento, M. L. Viana, and
C. J. P. de Lucena, “An iot-based tool for
human gas monitoring,” in IXV Congresso
Brasileiro de Informatica em Saude (CBIS),
vol. 1. SBIS, 2016, pp. 96–98.
[109] R. Morejón, M. Viana, and C. Lucena,
“Generating software agents for data min-
ing: An example for the health data area,”
in International Conference on Software En-
gineering & Knowledge Engineering-SEKE,
2017.
[110] S. Haykin, Neural Networks: A Com-
prehensive Foundation. Macmillan, 1994.
[Online]. Available: http://books.google.
com.br/books?id=PSAPAQAAMAAJ
[111] D. Castelvecchi, “Can we open the black
box of ai?” Nature News, vol. 538, no. 7623,
p. 20, 2016.
[112] L. Atzori, A. Iera, and G. Morabito, “The
internet of things: A survey,” Computer net-
works, vol. 54, no. 15, pp. 2787–2805, 2010.
[113] M. A. Salahuddin, A. Al-Fuqaha,
M. Guizani, K. Shuaib, and F. Sallabi,
21
“Softwarization of internet of things
infrastructure for secure and smart
healthcare,” Computer, vol. 50, no. 7, pp.
74–79, 2017.
[114] I. Ayala, M. Amor, L. Fuentes, and J. M.
Troya, “A software product line process to
develop agents for the iot,” Sensors, vol. 15,
no. 7, pp. 15 640–15 660, 2015.
[115] J.-P. Briot, N. M. de Nascimento, and C. J. P.
de Lucena, “A multi-agent architecture for
quantified fruits: Design and experience,”
in 28th International Conference on Soft-
ware Engineering & Knowledge Engineering
(SEKE’2016). SEKE/Knowledge Systems
Institute, PA, USA, 2016, pp. 369–374.
[116] N. M. Nascimento, “FIoT: An agent-
based framework for self-adaptive and
self-organizing internet of things applica-
tions,” Master’s thesis, PUC-Rio, Rio de
Janeiro, Brazil, August 2015.
[117] N. M. d. Nascimento, C. J. P. d. Lucena,
and H. Fuks, “Modeling quantified things
using a multi-agent system,” in IEEE / WIC
/ ACM International Conference on Web Intel-
ligence and Intelligent Agent Technology (WI-
IAT), vol. 1. IEEE, 2015, pp. 26–32.
[118] N. M. NASCIMENTO and C. J. P. LU-
CENA, “Engineering cooperative smart
things based on embodied cognition,” in
NASA/ESA Conference on Adaptive Hardware
and Systems (AHS 2017). IEEE, 2017.
[119] Apple, “Homekit,”
https://developer.apple.com/homekit/,
March 2017.
[120] Samsung, “Samsung smart things,”
https://www.smartthings.com, March
2017.
[121] D. I. Sjøberg, T. Dybå, B. C. Anda, and
J. E. Hannay, “Building theories in software
engineering,” Guide to advanced empirical
software engineering, pp. 312–336, 2008.
[122] S. Whiteson, N. Kohl, R. Miikkulainen, and
P. Stone, “Evolving soccer keepaway play-
ers through task decomposition,” Machine
Learning, vol. 59, no. 1-2, pp. 5–30, 2005.
[123] C. Wohlin, P. Runeson, M. Höst, M. C.
Ohlsson, B. Regnell, and A. Wesslén, Exper-
imentation in software engineering. Springer
Science & Business Media, 2012.
[124] R. S. Sutton and A. G. Barto, Reinforcement
learning: An introduction. MIT press Cam-
bridge, 1998, vol. 1, no. 1.
[125] R. Peck and J. Devore, Statistics: The Explo-
ration & Analysis of Data. Nelson Educa-
 22

tion, 2011.
[126] W. Oizumi, L. Sousa, A. Garcia, R. Oliveira,
A. Oliveira, O. Agbachi, and C. Lucena,
“Revealing design problems in stinky code:
a mixed-method study,” in Proceedings of
the 11th Brazilian Symposium on Software
Components, Architectures, and Reuse. ACM,
2017, p. 5.
[127] E. Fernandes, F. Ferreira, J. A. Netto, and
E. Figueiredo, “Information systems devel-
opment with pair programming: An aca-
demic quasi-experiment,” in Proceedings of
the XII Brazilian Symposium on Information
Systems on Brazilian Symposium on Infor-
mation Systems: Information Systems in the
Cloud Computing Era-Volume 1. Brazilian
Computer Society, 2016, p. 64.
[128] G. Kasparov, Deep Thinking: Where Machine
Intelligence Ends and Human Creativity Be-
gins. Hachette UK, 2017.
[129] D. Silver, A. Huang, C. J. Maddison,
A. Guez, L. Sifre, G. Van Den Driessche,
J. Schrittwieser, I. Antonoglou, V. Panneer-
shelvam, M. Lanctot et al., “Mastering the
game of go with deep neural networks and
tree search,” Nature, vol. 529, no. 7587, pp.
484–489, 2016.
[130] D. Silver, J. Schrittwieser, K. Simonyan,
I. Antonoglou, A. Huang, A. Guez, T. Hu-
bert, L. Baker, M. Lai, A. Bolton et al.,
“Mastering the game of go without human
knowledge,” Nature, vol. 550, no. 7676, pp.
354–359, 2017.
[131] A. Trotman and C. Handley, “Program-
ming contest strategy,” Computers & Edu-
cation, vol. 50, no. 3, pp. 821–837, 2008.