DevOps for AI – Challenges in Development of
AI-enabled Applications
Lucy Ellen Lwakatare Ivica Crnkovic
Computer Science and Engineering Computer Science and Engineering
Chalmers University of Technology Chalmers University of Technology
Gothenburg, Sweden Gothenburg, Sweden
llucy@chalmers.se ivica.crnkovic@chalmers.se
Abstract—When developing software systems that contain
Machine Learning (ML) based components, the development
process become significantly more complex. The central part of
the ML process is training iterations to find the best possible
prediction model. Modern software development processes, such
as DevOps, have widely been adopted and typically emphasise fre-
quent development iterations and continuous delivery of software
changes. Despite the ability of modern approaches in solving some
of the problems faced when building ML-based software systems,
there are no established procedures on how to combine them with
processes in ML workflow in practice today. This paper points out
the challenges in development of complex systems that include
ML components, and discuss possible solutions driven by the
combination of DevOps and ML workflow processes. Industrial
cases are presented to illustrate these challenges and the possible
solutions.
Index Terms—AI, Machine Learning, Agile software develop-
ment
I. I NTRODUCTION
ML is a rapidly developing and the most promising tech-
nology of the last decade, adopted in literally all business
domains and research areas. In many domains (e.g. medical
diagnose), ML has demonstrated superiority in providing re-
sults in comparison to traditional software applications, and
outperformed humans in the activities that require a human
intelligence. These trends have induced a huge need for data
and AI scientists, and also software developers. Early studies
[1] have shown that building algorithms (i.e. developing ML
model) is only a fraction of all efforts required to successfully
develop and put in operation an AI-based software system1 .
ML software implementations differ from traditional software
in that their logic is not explicitly programmed, but rather
automatically created by learning from data. As such, the
development process of ML-based systems involves different
activities, including data collection, data preparation, defining
the ML model (such as deep network model), perform the
process of training to quantify the model parameters and obtain
the expected result. This process is called ML workflow. ML
workflow requires sophisticated support of a set of tools and
The research in this paper has been supported by Software Center, the
Chalmers Artificial Intelligence Research Centre (CHAIR) and Vinnova.
1 We define AI-based systems as software systems that include components
developed using AI-modelling (e.g. ML components)
Authorized licensed use limited to: International Institute of Information Technology Bangalore. Downloaded on May 05,2021 at 10:48:20 UTC from IEEE Xplore. Restrictions apply.
Jan Bosch
Computer Science and Engineering
Chalmers University of Technology
Gothenburg, Sweden
jan.bosch@chalmers.se
activities, and to obtain an efficient process is a challenge of
itself.
However, ML workflow does not cover the entire software
development process. It does not address the question of
efficient software development, and that raises a question how
ML workflow and a software development process are related,
or as a more general question: which is the development
process of software systems that contain AI-components?
This paper discusses new requirements for software devel-
opment process that includes development of ML components.
It summarizes some new challenges that arise from the re-
quirements posed by ML components, and possible software
development process model that can be used to successfully
develop, operate and evolve ML based software systems.
The modern software development processes apply agile
principles, and one of the mostly commonly used agile pro-
cesses is DevOps where system development and its operation
are integrated in a common process which enables a fast feed-
back about the application performance and its usage to the
developers. Specifically, in this paper we discuss integration
of ML-workflow with DevOps.
The rest of this paper is organised as follows. Section III
gives an overview of ML workflow process with its main char-
acteristics. Section IV describes a DevOps process model with
its main characteristics. Section V discusses the problems and
challenges that occur in development of AI-based software,
requiring the integration of ML workflow and DevOps. In
section VI we describe possible ways of integration of ML
workflow and DevOps to successfully implement the system
life cycle. Finally section VII concludes the paper.
II. R ESEARCH METHODS
Our research is based on empirical evidence with companies
members of Software Center2 and Chalmers AI Research
Centre (CHAIR)3 including companies Ericsson, Siemens,
Volvo Cars, CEVT which are in a phase of extensive adoption
of ML in their development processes, and AI operation and
development companies such as Peltarion4 . Complementary
2 https://www.software-center.se/
3 https://www.chalmers.se/chair/
4 https://peltarion.com/
 Fig. 1. ML development workflow stages
research includes synthesis of multivocal ’grey’ literature, such
as blog posts, and our prior research [2], [3].
The AI-based systems we have explored in previous re-
search were largely in early stages. We refer to two cases
in Section VI.
• Case A: Out-Of-Office (OOO) Reply Detection. The OOO
reply detection system is an AI component within a
web-based sales engagement platform that is used to
optimize communication between sales representatives
and potential prospects. The AI component extracts in-
formation e.g., contact and date, from OOO email replies
and automatically prompts the sales rep to take relevant
action, such as add contact information, pause and resume
at the return date the automated sequence of sales actions.
• Case B: Fault Classification System in Returned Telecom-
munication Hardware. The AI component classifies prob-
lems (faults) in returned hardware based on log data from
radio networks. The screening operator, as an end-user,
connects the returned hardware to AI component and gets
a score that tells whether it is a software-related fault,
hardware-related fault or no fault. If the hardware is good
(has no fault), it is sent back to the customer but if it is
bad it is sent for repairing.
III. M ACHINE L EARNING WORKFLOW
Machine learning workflow describes the overall develop-
ment stages and activities that are typically performed when
developing ML based software systems. At high-level, a ML
workflow includes a set of distinguishable stages (see Figure
1): model requirements, data collection, data cleaning, data
labelling, feature engineering, model training, model evalua-
tion, model deployment and model monitoring [4]5 .
These stages can be grouped in separate processes: (i) Data
management, (ii) ML Modeling, and (iii) Model Operation.
Data Management process includes collection of data and its
preparation for its seamless usage. The Data Management
process can be performed separately from ML modeling - very
often data exists in enterprise repositories or as open repos-
itories, and they offer usage for different ML applications.
5 We define ML model as a ML function, or a ML component that will be
developed and deployed in a software system.
Authorized licensed use limited to: International Institute of Information Technology Bangalore. Downloaded on May 05,2021 at 10:48:20 UTC from IEEE Xplore. Restrictions apply.
The ML modeling process corresponds to the development
of a software component, or possibly a simple application -
i.e. creation of ML model itself, adjusting data, training, and
finally evaluating the model. This process is highly iterative
and driven by the results, such as the accuracy of the model,
the precision, or other quality attributes used during ML model
training. The deployment and monitoring of the trained ML
model belong to the operation process - integration of the ML
model in the software system and its performance. Monitoring
of ML component is seen as an integral part of a ML workflow
that is useful in providing feedback to ML modelling, which
may lead to a request for model re-training with new data sets,
or a redesign of the model itself with new features and new
model architectures.
In practice, the development of ML systems is challenging
for a variety of reasons [1], [5], [6]. Data that is used to build
ML models has numerous data quality problems and requires
huge efforts to prepare the data before training the models
because often it can not be readily fed into ML algorithms [7].
Furthermore, manual and ad-hoc procedures in ML modelling
stage make it difficult to reproduce model experimentation
results and reduces the rate of experimentation, including
tuning that can be done to improve the model [8], [9].
Despite its development challenges, ML model constitute
a small part of the entire ML system [1], [6]. Once data
scientists select the final ML model, it needs to be deployed
and integrated to the end-user application. For complex sys-
tems, the integration and deployment of ML models need
to take into consideration numerous requirements related to
the software system. The ML-based application also needs
continuous monitoring in order to detect errors in predictions
and data (e.g., training-serving skew) requiring that the ML
model be retrained due to staleness.
IV. D EVOP S PROCESSES
DevOps is a software development approach that empha-
sises collaboration between software development and opera-
tions in order to operate software systems and accelerate the
delivery of software changes [10]. The approach emerged to
be important in the context of continuous deployment (CD)
to help create a repeatable, reliable process for releasing soft-

Fig. 2. DevOps workflow
ware changes frequently in production environment [10], [11].
Building on agile software development method, an important
tenet in DevOps is automation of build, testing, deployment
and operations processes (Figure 2) while also ensuring that
all software artifacts are version controlled [10], [11]. As
such, DevOps practices in the deployment pipeline involve
automation of software deployment process, including auto-
matic provisioning of the environments aimed at minimising
manual hand-overs of the changes from software development
team to operations team. A deployment pipeline, in software
development practice, is a technical manifestation of the entire
software process constituting to all stages of getting software
changes from version control until they are visible to end-
users [11]. Automation in deployment pipeline, often done by
infrastructure teams, has the advantage to abstract the inner
workings of deployment procedure to software developers
who often experience disconnections between development
and production environments.
While the DevOps approach also has non-technical prac-
tices for effective collaboration between persons involved in
software development, much of the focus have been on the
technical practices. In online-based applications, deployment
mechanism is incorporated into the continuous integration
(CI) system with some defined triggers to facilitate the au-
tomatic deployment of changes to acceptance test or pro-
duction environments. As part of the deployment procedure,
configuration management tools are used to automatically
provision, configure and upgrade existing environments with
the new changes while applying a selected deployment strategy
e.g., blue-green deployment or rolling upgrade [10], [11]. In
other domains, such as Cyber-Physical systems, or embeeded
system, DevOps process include additional activities as the
operational environment is different from the development
environment. Simulations, when possible, are expensively used
to make the development cycles more effcient.
V. ML W ORKFLOW AND SOFTWARE SYSTEM LIFECYCLE
INTEGRATION CHALLANGES
The ML workflow (Figure 1) defines the end-to-end process
of building ML models, but it neglects software development
process. A deployed ML model is always a part of an
application or a software system. For example, in self-driving
vehicle systems, parts of driving tasks, such as environment
perception, steering wheel control and motion planning can
Authorized licensed use limited to: International Institute of Information Technology Bangalore. Downloaded on May 05,2021 at 10:48:20 UTC from IEEE Xplore. Restrictions apply.
be carried out by deep learning based approaches [12], but
this is only a fraction of the entire driving control system.
Development of software systems has another logic and a
different process (left part Figure 2). In modern development
processes operations are tightly connected with fast feedback
loops to the development process (right side of Figure 2),
but connection to ML workflow (data management and ML
modeling processes) are not well defined. That leads to a
number of challenges in performing both ML workflow and
DevOps. This section presents important challenges motivating
the integration of ML workflow and DevOps. The presented
challenges are based on our synthesis of scientific literature
[13] as well as experiences gathered from practitioners when
conducting empirical research and during reporting meetings
of Software Center and CHAIR center (see Section I).
1) Multiple environments: ML training assumes existence
of a storage with data that will be used in building ML
models. Most of the activities related to data processing can be
decoupled from other activities in the ML workflow. During
training, ML models are computationally demanding. Large
ML models require training across distributed high-performing
compute resources (e.g., graphical processing units (GPUs)) in
order to speed-up the training time. As such, the development
environment requires interactive software development tools to
schedule training on local or distributed computational units,
typically a combination of CPU and GPUs. On the other
hand, the operational environment can either be similar to
that of traditional software components (e.g. in web-based
applications), or completely different (e.g in vehicular systems
self-driving vehicle systems) because limited computational
and storage resources can involve offloading to the cloud.
Some of the major challenges of ML components, com-
pared to traditional software components, is how to deal with
versioned data and ML models, and not just versioned code.
Tools used in tracking and managing code changes, such as
Git in DevOps are inadequate to ensure efficient tracking and
management of versioned dependencies of ML artifacts (code,
data, models). Similarly, Continuous Integration (CI) tools are
not just about testing code but also for testing and validating
data that will be used to train models [14]. While practitioners
are observed to orchestrate ML model development using CI
tools [12], they are inadequate to ensure efficient running of
model training jobs over specialized hardware e.g., GPUs,
in the context of distributed training. In addition to different
tools, the processes require different expertise - data scientists,
AI-experts, and software engineers. These differences in the
environments, tools, processes and expertise required, pose
new challenges: a) How to ensure a successful management of
several development environments?, b) how to ensure a seam-
less intercommunication and exchange of artifacts between
them?, and c) How to ensure multidisciplinary expertise and
enable efficient collaboration between the domain-experts?
2) Complementary requirements: Modern software devel-
opment processes combine traditional requirements manage-
ment with outcome/data feedback from the stakeholders, and
recently a use of AI to enhance the feedback from the sys-
 tem operation [2]. Software system features related to users’
acceptance are typically evaluated through A/B testing. Non-
functional requirements related to different types of quality
attributes are in many domains the main concern of the
development process. Example of these are attributes related
to dependability (safety, security, reliability, robustness, etc.),
or resources (real-time requirements, performance, energy
consumption, computational and storage constraints), or user
interface (usability). The requirements related to ML workflow
are different: data quality, data management effectiveness (data
preparation, labeling), effectiveness of the training process
(CPU training time), and resources needed in the training
process. In relation to the quality of the ML models, there is a
set of metrics that show the quality of the prediction: accuracy,
precision and recall are some of them6 . In feature engineering,
the requirements are related a the quality and effectiveness of
feature selection and their processing (normalisation, transfor-
mations, etc.) in relation to the prediction quality.
In an integrated process, the main challenges lie in under-
standing the relations between software system requirements
and ML requirements. For example, relations between system
features and dataset features. Obviously these are not the
same features, but they might be related - when specifying
a feature as a system requirement, which features should
be included ML models? Another example: To achieve a
system reliability, do we need to ensure a certain level of
the accuracy of a ML model? There are similar problems
for the performance requirements: What is impact of ML
model performance to the system performance - for example
real-time requirements? These examples can be summarised
as a common challenge: a) How to relate ML requirements
to software system requirements?, and b) How to derive ML
requirements from the system requirements?
3) System Verification vs. ML Evaluation: The goal of the
ML evaluation is to increase the prediction quality - first to
find the minimum of the specified cost function, then provide
evaluation results in the form of accuracy and other ML
metrics. These metrics are unknown in the beginning and an
optimal value is searched for by changing the ML model and
input datasets in a number of iterations. This process does not
have predefined goals, nor estimations of efforts (in time and
used resources). Verification of a software system is somewhat
different - it is a measure of the correctness of the system
in relation to a specification [15]. The correctness can be
of a binary type (correct/failure) or a form of distribution
related to a non-functional attribute such as reliability. The
verification process is typically not an iterative process as
during the verification process the system is not supposed to
be changed/adapted. Such verification is typically realised by
testing, or by analysis (static or dynamic, formal or empirical).
Here again there is a question of how these processes are
related? Can we use the ML evaluation metrics to analyse
the correctness of the system? This question is in particular
relevant if a system has different AI models integrated in the
6 For definitions see https://en.wikipedia.org/wiki/Confusion matrix
Authorized licensed use limited to: International Institute of Information Technology Bangalore. Downloaded on May 05,2021 at 10:48:20 UTC from IEEE Xplore. Restrictions apply.
system. Or the opposite, can we from the system verification
come to a conclusion that a ML model is not sufficiently
accurate or precise? The challenge: How the software system
verification is related to ML evaluation?
4) System evolution vs. ML model evolution: A continuous
change of software has led to the development processes like
DevOps, which explicitly support continuous integration and
continuous deployment, automatic verification and specific
type of testing in production, such as A/B testing. Software
evaluation includes changes of new and improved functional-
ity and internal infrastructural change. These changes have
impact on requests for changes of ML modules, including
both functional and non-functional properties. ML workflow
focuses on another type of evolution: a) evolution because of
re-training needs caused by changes of datasets. New data
originated from new contexts may require re-modeling by
re-training; b) evolution related to changes/optimisation of
the model itself but trained on the same datasets. While the
continuous software evolution can be mapped to the evolution
of ML models in a controlled way (though that requires
established procedures), change of a context, i.e. changes of
data input for ML models, can happen in an uncontrolled and
unexpected manner, which might have direct impact on ML
model accuracy and performance. The context change may
require ML Model retraining with updated data sets. However,
a retrained model can work in the new context correctly, but
its performance degrades for the old context. The challenges
that arise are the following: a) how does software evolution
influence the ML model evolution?, b) how to control the
operational context change and its influence on ML model
performance and a need for ML models evolution?, and c)
how to control evolution of ML models to achieve a same
level of control of software system evolution?
5) Operations and Feedback loops: ML workflow and
DevOps are designed as iterative processes themselves, but
they also have mutual dependencies. Data management in-
cludes a data collection activity, and data must be collected
to be available for the training in the initial phase of the
development. However, in many cases new data is collected
during the system operation, in particular if new data is needed
to cover new operation contexts, and the ML model must be
retrained. A ML Model must be deployed in the development
environment to be used in the development process, and
deployed together with the executable system in the opera-
tion. In the operation, the system must provide information
about the system performance and users’ acceptance to the
development environment in order to provide a feedback for
further system evolution. The system also needs to monitor
deployed ML models in order to check their performance in
the current context. There is information of different type that
must be continuously exchanged for a successful and efficient
end-to-end process. The challenge that appears here is: a)
which information should be exchange between the processes?,
and b) what are the triggering mechanisms for information
exchange and in which frequencies the information should be
exchanged?
 Fig. 3. ML workflow and DevOps process integration
VI. ML W ORKFLOW AND D EVO PS P ROCESS
I NTEGRATION
Ultimately, the integration of ML workflow and DevOps
processes should enable fast, iterative and continuous
development, deployment and operation of AI-based software
systems in production. The integrated process advocates
automation within and across all the process steps of AI-
based system implementation, including building, testing,
deployment and infrastructure management. All artifacts
(code, data and models) important to construct AI-based
system are reproducible and can be adapted and released
reliably in small increments. The integrated process especially
applicable in scenarios wherein deployed ML models are
retrained based on new streamed data or optimization of ML
models parameters, such as changing hyper parameters. The
entire AI-based system life cycle integrates ML Workflow
and DevOps into four distinct processes, Data Management
(DM), ML modeling (Mod), Software Development (Dev), and
System Operation (Ops) (see Figure 3).
DM. Data management stage establishes activities and
systems to collect data, select data, augment labelling and
curate features that will be used for training ML models.
Standardized activities and systems are put in place to help
extract and store data in optimized formats that are easy and
ready to be consumed by ML modelling stage, while also
enforcing suitable security controls for accessing the data.
The resulting data set building code creates connectors to raw
data storage in order to select and extract samples of data
from them. In Case A, all email communication data between
sales representatives and prospects is stored in communication
database from which a few data samples are selected and
labelled. However, the extraction involves a series of steps to
handle different encoding in order to get email text, extract
entities and construct relationship around the entities. In Case
B, a data ingestion service is implemented to build data set by
extracting data from a large data collection storage containing
log files and log analysis (performed by an external tool).
Mod. Once new data set is made available for training
Authorized licensed use limited to: International Institute of Information Technology Bangalore. Downloaded on May 05,2021 at 10:48:20 UTC from IEEE Xplore. Restrictions apply.
at ML modelling stage, training environment is provisioned
to facilitate model training and evaluation. ML model
experiments can be carried out on ML platform, developed
either internally or acquired as open-source tools, that
provides the capability of analysing data sets and training
ML models. At the stage information of each ML model
experiment run i.e., datasets, trained model (including model
configurations) and evaluation results are stored for easy
comparison against a historical baseline e.g., a model that
is currently in production. To facilitate tracking of artifact
dependencies after a successful completion of ML model
experiment, developers explicitly specify and register artifacts
and their dependencies (data set, model and code) to a
dependency tracking system that stores the experiment run
history in a database. For each artifact, the developers specify
its metadata (name, version number, registration date) and
dependency information, which contain references to other
elements that the artifacts depend on. For example, data set
dependency information can be (i) data source e.g., log IDs,
(ii) Git hash of the code used to extract data set from raw
data, (iii) versioned set of data labels and (iv) entry point
to extraction scripts etc [12]. Important to storing artifacts
and their dependency information is that it should allow
immutability i.e., it should be possible to recreate/reproduce
the exact same artifact every time. The versioned artifacts and
their dependency information is made accessible externally
by other systems e.g., by build system through simple API
calls. By the end of model experiment stage, the final trained
model is selected and verified that it works correctly but
in isolation from other ML components. Case A uses an
externally developed AI platform to build and deploy models
called Databricks With the tool, Case A stores the whole
pipeline of conducting ML model experiment steps as a
single archive file and the resulting models in a versioned
controlled repository. In Case B, the ML modelling stage
involves manual steps of data exploration, model training and
evaluation. Developers implement and version model training
code after successfully training a model. The model along
with metadata information (name, version, classifier, features)
is also stored in model registry.
 Dev. When deploying the trained model, first it is verified
that it operates correctly with the rest of the system i.e.,
other components. The build system performs system wide
integration and verification over large test set to give holistic
view of how all parts of the system performs between the
different components. Additionally, since artifact dependency
information and capability is accessible to build systems,
such as the versioning and storing artifacts using code (data
set generation code, model training code, evaluation metrics
generation code), the build system can be implemented in
such a way that it has the functionality to orchestrate and
run the entire model training job on a regular basis. In case
of the latter, the retraining models is automated and process
is designated as failed, aborted or successful by the build.
Successful build with best model - new model performs
better than production model - is registered in model registry.
In case of failed and aborted builds, developers can opt
to rebuild the model with an optimized set of parameters.
Historical builds executed by the CI system can be used to
also identify and analyse bugs that occur in other components
as a result of model updates. At the end, the verified latest
registered version of a trained model is released/promoted
to the subsequent quality assurance acceptance environment
or the production environment only after successful build,
including passing of regular integration and system tests. The
regular tests include testing the AI component with different
inputs to ensure appropriate response. In case A, one can
reproduce the trained models by re-running the single archive
file containing the whole model pipeline steps generated by
Databricks tools. At the time of our study, both Case A and
Case B were in the process of implementing tests. Case B
was also planning to incorporate a CI system in order to help
deal with technical debt that was accumulating due to time
pressures requiring the team to delivery the system fast to the
stakeholder. For example, unit tests were being implemented
to help catch issues caused by the data ingestion service.
Ops. Once the ML system component is operational in pro-
duction environment, including the trained model, the system
is continuously monitored to detect issues like performance
degradation. Deployment to the production environment can
be a manual, semi-automated or automated process that is
performed after successful execution of tests in pre-production
environments in development step. The Ops stage also ensures
live testing of AI component by providing predictions whilst
adhering to stringent latency requirement. At this step, devel-
opers also collect information of how the model is performing
based on live data. This information can be used to trigger
retraining of models. Both Cases A and B have capability to
monitor how the model is performing in production.
VII. C ONCLUSION
In this paper we have illustrated the integration of ML
workflow and the DevOps process as an approach to help build
AI-based systems systematically. The approach also helps to
tackle some of identified challenges when developing ML-
Authorized licensed use limited to: International Institute of Information Technology Bangalore. Downloaded on May 05,2021 at 10:48:20 UTC from IEEE Xplore. Restrictions apply.
based software systems. In particular, the manual steps during
ML model experiment and deployment including addressing
the problems of version control and dependency management
for ML artifacts. This paper tries to energize and motivate
further discussions in addressing the many challenges in
the development of AI-based systems. In future work we
will elaborate more on ML workflow and DevOps process
integration with additional cases.
R EFERENCES
[1] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner,
V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden
technical debt in machine learning systems,” in Advances in neural
information processing systems (NIPS) 28. Curran Associates, Inc.,
2015, pp. 2503–2511.
[2] J. Bosch, H. H. Olsson, and I. Crnkovic, “It takes three to
tango:requirement, outcome/data, and ai driven development,” in Inter-
national Workshop on Software-intensive Business: Start-ups, Ecosys-
tems and Platforms). http://ceur-ws.org/, 2018, pp. 177–192.
[3] L. E. Lwakatare, A. Raj, J. Bosch, H. H. Olsson, and I. Crnkovic,
“A taxonomy of software engineering challenges for machine learning
systems: An empirical investigation,” in Agile Processes in Software
Engineering and Extreme Programming, P. Kruchten, S. Fraser, and
F. Coallier, Eds. Springer International Publishing, 2019, pp. 227–243.
[4] S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagap-
pan, B. Nushi, and T. Zimmermann, “Software engineering for machine
learning: A case study,” in 41st International Conference on Software
Engineering: Software Engineering in Practice. IEEE, 2019, pp. 291–
300.
[5] M. Kim, T. Zimmermann, R. DeLine, and A. Begel, “Data scientists in
software teams: State of the art and challenges,” IEEE Transactions on
Software Engineering, vol. 44, no. 11, pp. 1024–1038, 2018.
[6] A. Arpteg, B. Brinne, L. Crnkovic-Friis, and J. Bosch, “Software
engineering challenges of deep learning,” in 44th Euromicro Conference
on Software Engineering and Advanced Applications, 2018, pp. 50–59.
[7] N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, “Data
management challenges in production machine learning,” in Proceedings
of the 2017 ACM International Conference on Management of
Data, ser. SIGMOD ’17. New York, NY, USA: Association for
Computing Machinery, 2017, p. 1723–1726. [Online]. Available:
https://doi.org/10.1145/3035918.3054782
[8] C. Hill, R. Bellamy, T. Erickson, and M. Burnett, “Trials and tribulations
of developers of intelligent systems: A field study,” in Symposium on
Visual Languages and Human-Centric Computing. IEEE, 2016, pp.
162–170.
[9] S. Schelter, J.-H. Böse, J. Kirschnick, T. Klein, and S. Seufert, “Au-
tomatically tracking metadata and provenance of machine learning
experiments,” in NIPS Workshop on Machine Learning Systems, 2017.
[10] L. E. Lwakatare, T. Kilamo, T. Karvonen, T. Sauvola, V. Heikkilä,
J. Itkonen, P. Kuvaja, T. Mikkonen, M. Oivo, and C. Lassenius, “Devops
in practice: A multiple case study of five companies,” Information and
Software Technology, vol. 114, pp. 217 – 230, 2019. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0950584917302793
[11] J. Humble and D. Farley, Continuous Delivery: Reliable Software Re-
leases through Build, Test, and Deployment Automation, 1st ed. Boston:
Addison-Wesley Professional, 2010.
[12] Y. Guo, K. Ashmawy, E. Huang, and W. Zeng, “Under the hood
of uber atg’s machine learning infrastructure and versioning control
platform for self-driving vehicles,” Mar 2020. [Online]. Available: https:
//eng.uber.com/machine-learning-model-life-cycle-version-control/
[13] L. E. Lwakatare, A. Raj, I. Crnkovic, J. Bosch, and H. H. Olsson, “Large-
scale machine learning systems in real-world industrial settings: A re-
view of challenges and solutions,” Information and Software Technology,
vol. 127, 2020.
[14] Google, “Mlops: Continuous delivery and automation
pipelines in machine learning,” April 2020. [Online].
Available: https://cloud.google.com/solutions/machine-learning/
mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
[15] C. Murphy, G. E. Kaiser, and M. Arias, “An approach to software testing
of machine learning applications.” in SEKE, vol. 167, 2007, pp. 52–57.