Model-Driven Continuous Deployment for Quality DevOps

Matej Artač Tadej Borovšak Elisabetta Di Nitto

XLAB XLAB Politecnico di Milano
Pot za Brdom, 100 Pot za Brdom, 100 Via Golgi 42
Ljubljana, Slovenia Ljubljana, Slovenia Milan, Italy
matej.artac@xlab.si tadej.borovsak@xlab.si elisabetta.dinitto@polimi.it

Michele Guerriero 27/5/2016 Damian A. Tamburri draw.io

Politecnico di Milano Politecnico di Milano

Via Golgi 42 Via Golgi 42
Milan, Italy Milan, Italy
michele.guerriero@polimi.it damianandrew.tamburri@polimi.it

ABSTRACT Modeling Environment

DevOps entails a series of software engineering strategies TOSCA

and tools that promise to deliver quality and speed at the model
same time with little or no additional expense. In our work
we strived to enable a DevOps way of working, combining Infrastructure ATL Deploy 
Transformation
Xtext Roll-out 
Model-Driven Engineering tenets with the challenges of de- Engineer

livering a model-driven continuous deployment tool that al- TOSCA 

MODAClouds4DICE
lows quick (re-)deployment of cloud applications for the pur- model
YAML 
Blueprint
pose of continuous improvement. This paper illustrates the DEV
DICER tool and elaborates on how it can bring about the
DevOps promise and enable the quality-awareness.
OPS
Front-end
service
CCS Concepts
Cloudify Chef Server
•Software and its engineering → Model-driven software
engineering; Extra-functional properties;
Storm Zookeeper Hadoop
Cookbook Cookbook Cookbook
Keywords Technological Library

Model-Driven Engineering; Continuous Deployment; Quality-Aware Deployment Service

DevOps;
1. INTRODUCTION
The software engineering approach known as DevOps entails
using cooperation, coordination and joint technical efforts and
tools shared between development and operations people. This
approach reduces the time a design change takes to get into pro-
duction value [4]. Efficient DevOps is therefore a mix of organ-
isational software culture [4] as much as supporting the DevOps
strategies in place with the proper tooling. As tooling becomes
paramount, e.g., in scenarios where speed and agility is key such
as Big Data [3], there are many problems that both newcom-
ers and the experts in the field face when working with multiple
application frameworks or middleware, e.g., the many Big Data
technologies currently on the market [6, 1]. On one hand, techni-
cians require a considerable skill-set to set up the needed services,
connect them together and perform this consistently throughout
the development. On the other hand, appropriate tooling needs
to be provided to enable the coherent and joint evolution of de-
chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html
Figure 1: DICER framework, Architecture
Overview.
signs and deployments based on the evolving needs of the appli-
cation. In this paper we propose a framework called DICER that
supports the model-driven continuous design and deployment, in
a DevOps fashion, of modern data-intensive applications. Our
research solution also enables quality-aware DevOps approaches,
since it gives support to many possible analysis. DICER is built
combining a Modeling Environment based on the TOSCA1 stan-
dard orchestration language and a Deployment Service based on
Cloudify and Chef. The adoption of the TOSCA standard enables
treating deployment as code following the DevOps tenets [4]. It
is worth to note that both Cloudily and Chef has been chosen
to implement the Deployment Service architecture, but this is by
itself independent from both of them.
Both the Modeling Framework and the Deployment Service
that are part of the DICER framework are integral parts of the
DICE IDE [2] too, in the context of the DICE EU H2020 project2 .

Permission to make digital or hard copies of all or part of this work for personal or 2. A MODEL-DRIVEN CONTINUOUS DE-
Permissionuse
classroom to is
make digital
granted or hard
without fee copies
provided of that
all orcopies
part of
arethis
not work
madefor personal or
or distributed
classroom
for profit oruse is grantedadvantage
commercial without fee provided
and that copies
that copies arenotice
bear this not made or distributed
and the full cita- PLOYMENT TOOL
for profit or commercial advantage and that copies bear this notice
tion on the first page. Copyrights for components of this work owned by others and the full citation
than Figure 1 depicts our solution architecture. It comprises two
on the first page. Copyrights for components of this work owned by others than ACM
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- main components, the Modeling Environment and the Deploy-
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
publish, to servers
to post on post on or
servers or to redistribute
to redistribute to lists, to lists, requires
requires prior specific
prior specific permission
permission and/or a ment Service. Through the Modeling Environment Dashboard a
and/or a fee. Request
fee. Request permissions
permissions from permissions@acm.org.
from Permissions@acm.org. user (e.g., an infrastructure engineer) can drag and drop deploy-
QUDOS 2016
QUDOS’16, Saarbrucken,
July Germany Germany
21, 2016, Saarbrücken, 1

cc 2016
2016ACM.
ACM.ISBN 123-4567-24-567/08/06.
978-1-4503-4411-1/16/07...$15.00 http://docs.oasis-open.org/tosca
2
http://dx.doi.org/10.1145/2945408.2945417
DOI: 10.475/123 4 http://dice-h2020.eu/

40
ment elements and properly link and configure them according to
the MODAClouds4DICE meta-model for the deployment spec-
ification of data-intensive applications. In this work, when we
refer to deployment model of a data-intensive application, this
includes the deployment of all the required services, technolo-
gies and of the jobs to be run over resulting platforms. To build
MODAClouds4DICE, we extended MODACloudsML [5], which
is a modeling language to specify the provisiong and deployment
of multi-Cloud applications. In particular we defined a set of con-
figurable nodes specific to Big Data technologies such as Apache
Hadoop or Apache Storm. We also enabled configuring client
nodes, which can submit the user’s custom workfows or jobs on
top of the Big Data techonologies, making the framework suit-
able for modeling data-intensive applications. Once the user has
drafted her initial model, this is transformed into a TOSCA-
compliant model using a model-to-model transformation, devel-
oped in the ATLAS Transformation Language (ATL), called the
Deploy Transformation. This transformation is designed over the
MODAClouds4DICE and the TOSCA meta-models. By extend-
ing the MODACloudsML language and using it on top of TOSCA,
the users can easily specify deployable Cloud-based deployment
models. The TOSCA model output of the Deploy Transforma-
tion is finally serialized, using an Xtext model-to-text transfor-
mation, into a deployable TOSCA YAML blueprint, which is in
turn automatically sent to the Deployment Service in order to be
processed.
The Front-end service component of the Deployment Service
receives TOSCA blueprints via RESTful APIs. These APIs pro-
vide an abstraction of the specific cloud orchestrator, Cloudify.
The cloud orchestrator, in turn, leverages the Chef tool to per-
form technoloy installation and configuration. Both Cloudify and
Chef rely on the content part of the framework represented by the
Technology Library, which consists of Chef cookbooks and dec-
larations of the supported technologies and the recipes for their
deployment, installation and configuration. It is worth to note
that the nodes a user can instantiate while using the Modeling
Environment strictly reflect the TOSCA node types defined in
the Technological Library of the Deployment Service. As a re-
sult, the Modeling Environment can produce blueprints, which
declare only the topology of the application and some deploy-
ment configuration parameters, while all the deployment steps
implementation is hidden away.
The Deployment Service can be operated using its simple web
user interface, but the principal use of the tool is to be integrated
in a larger DevOps workflow. Normally, it will be invoked from a
Continuous Integration tool (e.g., Jenkins or Apache Continuum).
The service operates with uniquely identifiable logical unit in the
deployment environment, simply called virtual containers 3 . A
container can receive an application blueprint expressed in stan-
dard TOSCA YAML format. An administrator will normally
create one or more empty containers and share their identifiers
with the developers, or use them to associate containers with spe-
cific Continuous Integration jobs. When a user or a client sends a
blueprint to be deployed in an empty container, the Deployment
Service will deploy the Big Data technologies according to the
specifications in the TOSCA blueprint, and launch the user’s job
within the deployment. If a client later submits another blueprint
to the same container, the tool will first assure that the previous
deployment has been purged.
3. DISCUSSION AND CONCLUSIONS
Supporting DevOps efficiently means providing tools and me-
thodologies that harmonise and speed up the work of software
designers, developers and operators as a joint and cohesive work-
force. In providing said effective support, in previous works we
outlined a software architecture that accelerate the design and
deployment of data intensive applications [1, 6].In this paper we
present the DICER tool, which focuses on the actual deploy-
ment and represent a contribution toward applying DevOps to
modern data-intesive applications, bringing a model-driven ap-
proach into a fully functional continuous deployment workflow.
3 IEEE, 2016.
Term not related to Docker or any similar technology.
41
This is achieved through two main results: on one hand MODA-
Clouds4DICE provides a language for the deployment of data-
intensive applications; on other hand the Technological Library
contributes to TOSCA by defining data-intensive node types based
on Chef cookbooks. Moreover DICER open the way for quality
DevOps: since different kind of analysis can be performed on top
of our model-driven approach, while the Deployment Service can
support the enforcement of runtime quality assurance methods
(for instance by integrating with specific enhancement tools) and
the implementation of feedback mechanisms to keep updated the
upstream models with monitoring information.
We designed the Deployment Service to use containers in order
for the tool to support a tidy and well organized approach to
continuous deployment. This means that there would be only as
many concurrent blueprint deployments in the test bed as there
are containers available. The tool then takes care of cleaning up
after the deployments when they become superseded with new
ones. A container may, for example, represent a specific branch
in the development of a data-intensive application. When the
developers update the design and the code of their application,
they receive a consistent deployment, which is always associated
with their application’s latest version.
To experiment with DICER, we created a minimal Storm blue-
print, consisting of one Zookeeper node, one Storm Nimbus node
and three Storm worker nodes using the Modeling Environment.
We then used our deployment tool on top of the Flexiant Cloud
Orchestrator (this is not the only supported cloud. Among the
others, we support also OpenStack and AWS), and the deploy-
ment took 30 minutes on average. This is much faster than hours
or days in a manual scenario.
We identified several key limitations that may require further
attention and research. One of them is that the speed-up of
deployment may still not be practical enough for frequent de-
ployments and tests. Another down side is that the current im-
plementation of the Modeling Environment and the Deployment
service enable submission of Big Data jobs (e.g., a specific Storm
topology or a Spark application) as the only kind of users’custom
applications. Components such as web services are not supported
yet. Similarly, the tools only use the Chef cookbooks packed in
the Technology Library, while the users’custom cookbooks and
recipes are not yet supported. We will address this in the upcom-
ing releases of the DICER framework. In the future we plan to
empirically strengthen the validity of our framework and solution
architecture stressing it with heavyweight industrial case-studies
as much as action research. Also, we plan to prioritise our ten-
tative research roadmap by means of industrial focus-groups so
that the key research venues from an industrial adoption and ex-
ploitation perspective may be investigated first.
4.
[1]
REFERENCES
M. M. Bersani, F. Marconi, D. A. Tamburri, P. Jamshidi,
and A. Nodari. ”continuous architecting of stream-based
systems”. pages 115–121, 2016.
[2] G. Casale, D. Ardagna, M. Artac, F. Barbier, and E. D. N.
et Al. Dice: Quality-driven development of data-intensive
cloud applications. In Proceedings of the 7th International
Workshop on Modelling in Software Engineering (MiSE),
May 2015.
[3] M. Chen, S. Mao, and Y. Liu. Big data: A survey. MONET,
19(2):171–209, 2014.
[4] C. A. Cois, J. Yankel, and A. Connell. Modern devops:
Optimizing software development through effective system
interactions. In IPCC, pages 1–7. IEEE, 2014.
[5] N. Ferry, H. Song, A. Rossini et Al.. Cloud mf: Applying
MDE to tame the complexity of managing multi-cloud
applications. In Proceedings of the 2014 IEEE/ACM 7th
International Conference on Utility and Cloud Computing,
UCC ’14, pages 269–277, 2014. IEEE Computer Society.
[6] M. Guerriero, S. Tajfar, D. A. Tamburri, and E. D. Nitto.
”Towards a Model-Driven Design Tool For Big Data
Architectures”. In proceedings of the 2nd International
Workshop on Big Data Software Engineering - BIGDSE.