Systematic Literature Review on:

DevOps Implementation Strategy

Zidni Nurobbi Agam, 1901726892
Nur Hakim Arif, 1901726785
Benni Bangun, 1901727062

Abstract
DevOps is a method of approach where roles in software engineering are merged with a mix of
pattern with purpose to improve collaboration between development and operations with expected
outcomes to improve the production release frequency and maintain software quality. We can say
DevOps is a paradigm or culture where development and operational team are collaborating to fulfil
organization needs.
The purpose of this study is to do an analysis on impact of DevOps whether if it has benefits to
organization. The main objective of this research is to identify whether there is a relationship
between effort to business, measuring benefits and responsiveness to business needs, know how
the implementation of DevOps can adapt quickly and easily to business needs, identify the success
factor for implementation DevOps. Other objectives of this research are to gather how to design
change management to implement DevOps.

Introduction
In the organization that develop and use Information System technology, they make their own
structural division/department of IT [1]. This division generally divided into Software Development
and System Operations. Software Development division develop a new software based on the
organization needs while the System Operations division maintain the software on daily basis. Many
discussions were questioned whether this kind of separation is necessitated. In recent years a new
concept has been emerged, DevOps. DevOps is a one of the newest method for organization to
improve some efficient and effective environment for the development and operation teams [2].
DevOps aims at extending the agility of the software development team to the operations team the
latter being responsible for deploying, monitoring and supporting the system.
Although DevOps have many advantages, there are challenges to adopting and implementing this
method in the organization [3]. The adoption challenges for DevOps implementation such as trying
to convince the owner that the method used will be useful for the organization, get a support from
complex enterprise environment, maintain the teams that will migrate application to DevOps
environment because the migration require a strong period of time with long duration [4]. To
implement this method the organization need to be well prepared and create strategy to face the
challenges. The organization also has to differentiate behavior and culture from traditional method
to DevOps method [5].
Adoption of DevOps to replace existing will affect to all aspect in an organization. By implementing
DevOps structural department of IT will change, in a large enterprise little changes in organization
structure will make a big change on perspective on responsibility and measurement [6]. Another
related issues in DevOps adoption is budgeting, DevOps need further tools to assure that method
running smoothly.
This paper are arranged by Systematic Literature Review (SLR) conducted by [7]. This research will
gather all literatures that needed in order to implement DevOps concept on an existing organization
that already separates its department to Development and Operation departments. This study
explores the methods to implement DevOps and measure the readiness of organization for adopting
DevOps. This paper also describes the sucess factor in implementing DevOps and how to design the
change management for it.

Related Works
DevOps terminology arise and supported by other concepts and methods. This is because modern
software engineering require a very fast and flexible development strategy [2]. This ability makes
engineering team adapt rapid and significant change while maintaining projects. To elaborate the
complexity of software engineering, the methodology will be simplified. This section will describe
the available methodology of software engineering and how to combine those methods to build a
DevOps team. In the end of this section, a brief of state of the art of systematic literature review will
be explained.
Software Development Life Cycle
Software Development Life Cycle (SDLC) is a model that conceptualize a software development
phases, i.e.: Planning, Designing, Testing and Deploying [2]. Waterfall model commonly used in this
methodology. Waterfall model views the processes as a sequence that flow down moving downward
on project completion. The waterfall model can be implemented when risks tightly controlled and
changes on project restricted after the requirement defined [8]. In a large-scale systems project
where complexity and cost are so high, the Waterfall approach provide rigid steps that helps to
ensure careful completion of all deliverables in every phase. Figure 1 show an example of Waterfall
methods.
Analysis

Design

Development

Testing

Deployment

Figure 1. Waterfall model diagram

Another models that developed based on refinement of waterfall model is Spiral model. This model
applied on large government projects and open to changes and revisions after a life cycle finished
and return to requirement phase. This approach is suitable for projects where changes can be blend
with reasonable cost increases or with acceptable time delays [8]. Figure 2 illustrate the Spiral
model.

Figure 2. Spiral Model

Agile Methodologies
Agile is a framework for developing software that emphasize on quick development in iterative
steps. It focus on personal communications among developers, users and testers in same or nearby
rooms [9]. Agile development depends on less written documents than traditional development
methods. This method is focus on quick adaptation to business changes. Agile methods work best in
“emergent” requirement environment where business conditions include ambiguity. This method
less effective is systems where safety, extreme accuracy or critical security features must be present,
e.g.: NASA or nuclear energy industry.
Some popular tools for Agile Methods i.e.: Scrum, Kanban or Extreme Programming provide a
process model to develop products. Agile methodology lack of definition of the right kind of product,
as a trade-off this methodology offers customer needs and expectation fulfillment in very short time
[10]. Agile methods is a response of waterfall drawbacks in similar change-resistant software
development methodology. Practitioner believe that designing processes to adapt changes and
based on new information effectively and efficiently leads to reduced project risks, and the other
side it will significantly enhanced project outcomes [2]. To constantly adapt the changes, teams
require highly effective communication ability. These teams often leverrage specialized tools and
techniques to ensure rapid and robust communication both within teams and between teams and
stakeholders.
The distinct differences between agile methods and traditional methodology are on its speed of each
iterations [9]. Agile measured in weeks rather than months. This method will separate tasks in a
project into single-executable tasks, those tasks will split up into repetitions. Every repetition has
their own analysis, objectives, testing scheme and deployment plan. Iteration model in agile shown
in Figure 3

Figure 3. Agile Method Model

Data-Driven Development
Project progress should be monitored in software development projects. Degraded progress in
project should be detected swiftly to allow management to adjust its processes and handle the
issues. Data-driven project management technique have gained popularity because of collection of
accurate and up-to-date information in a project [2]. Constant collection of large amounts of data
can be used to calculate informative metrics on project status. These metrics can be used to guide
project management decisions, adjust the project plans and forecast team performance. Though
data-driven processes offer measurable gain in performance. All aspect will interpret as a metric
data and related one and each other. To achieve the advantages of this method, automated system
should be conducted to collect data without direct assistance of human actors.
DevOps
DevOps has evolved in response to the lack of collaboration efforts between the development and
operations teams [11]. The culture of DevOps has been acting as an enabler to deliver more features
recurrently while maintaining stability. This continual process has led to continuous delivery which
deliver the product to the production environment without affecting or has small effect in the
production itself.
DevOps aims at extending agility of the software development team to the operation teams and
latter will be responsible for deploying, monitoring and supporting the system [3]. Three factors that
led DevOps rapid adoption increasing, i.e.: 1) Higher quality expectation from software as it is
increasingly offered as a service in the cloud, 2) Demand for rapid delivery of change with growing
acceptance of agile and, 3) The availability of on-demand powerful and plentiful hardware on the
cloud. Various tool sets and research in DevOps has given rise to new features for infrastructure
automation in terms of stability and agility. With evolving model of continuous delivery, there has
been significant contribution to standardization of processes and has led to unification and
agreement of objectives between teams [11].

Methodology
Research Questions
A systematic literature review purpose is to identify, evaluate and interpret all available research
relevant to a particular research question or topic area [7]. The common reasons performing
systematic literature review i.e.: 1) Summarize existing evidence, 2) Identify any gaps in current
research, 3) Provide a framework to positioning new research activities.
This study aims to gather the state of the art of the literature that related to DevOps
implementation. Therefore, the main question of this research is: “How to implement DevOps on an
existing organization?”, this main research question will inherit three research questions (RQ):
RQ. 1. What the most impact of DevOps to organization?
This RQ aims to explore the result of DevOps implementation in a perspective of an organization.
All the impacts yielded by DevOps Implementation, whether positive impact or negative impact
will be captured

RQ. 2. How to measure organization readiness for DevOps adoption?

A successful DevOps implementation require a thorough readiness in an organization. This RQ
will answer the organization readiness measurement to implement DevOps.

RQ. 3. What is the success factor in DevOps initiatives?

 Concerning this RQ, this study will analyze the key factor that brought DevOps implementation
success. This RQ analyze the literatures based on their implemented proposed system.

RQ. 4. How to design change management for DevOps implementation?

Change management strategy needed when an organization migrate their running environment
to a particular system. With a proper change management strategy, the migration will be
running as planned. In terms of this RQ, this study will propose a framework to design change
management strategy for implementing DevOps.

Research Methodology
This research applied method in [7]. This research was conducted in 3 phases and divided into 7
tasks, i.e.: 1) Planning, 2) Piloting the articles, 3) Searching in repositories, 4) Inclusion and Exclusion
of articles, 5) Quality Assessments, 6) Analysis and clustering the studies, 7) Reporting the SLR. The
all phases can be shown in Figure 4.

Phas Planning
e1 Piloting the articles
Plan
ning
Searching in Repositories
Phas Inclusion and Exclusion of articles
e2 Quality Assessments
Cond Analysis and Clustering the studies
uctin
g

Reporting The SLR

Phas
e3

Figure 4. Research Methodology Diagram

At the first phase, this study will formulate the problems and arrange research plans. In Planning
task, the research will determine the problem statement, RQ and research scope. In the Piloting
task, research will conduct the terms that will be used in gathering articles from repositories. In
addition, this task will specify repositories that used in this research. This phase will also return a
framework for the next phase.
In the first phase, several common research repositories in IT management implementation was
used, the repositories are: ScienceDirect, IEEE Xplore, ACM Digital Library. Literatures that
considered in this research were: 1) 5 years old paper or less; 2) has relations in problems that
overlap in software development and software operations; 3) considering in DevOps
implementations.
Conducting Phase will perform after the Planning phase by research groups. By using the keyword
and other search parameters that have been determined, Searching in Repositories task carried out
simultaneously by researchers. In this study some keywords were used to search the articles in
repositories, i.e.: The keywords applied on those repositories are: DevOps AND impact, DevOps AND
readiness, DevOps AND “success factor”, DevOps AND “change management”. Other options were
applied to the repository’s search engine, i.e.: 1) Published in 2013 up to 2018; 2) Categorized in
Computer, Software Development or Software Operations; 3) Other keywords: DevOps
implementations. Repositories used in this study were: 1) ScienceDirect; 2) IEEE Xplore; 3) ACM
Digital Library. The total of articles resulted in this step were shown in Table 1 below.
Table 1. DevOps Articles/Papers from Repositories

Repositories Total
IEEE Xplore 49 Articles
ScienceDirect 32 Articles
ACM Digital Library 29 Articles

The next task is Inclusion and Exclusion the Articles, papers from repositories were selected by
highlighting their titles and abstract, papers that not related to the study will eliminated briefly. The
quantity of selected articles in this task will be considered, if the total of eliminated articles is more
than nine-tenth of all articles then the previous task will be performed with broaden the keywords
and parameter. This step to assure keywords that used are relevant to the searching method.
In the Quality Assessment Task, the rest of the papers then will ranked based on the paper quality
[7], ranking of the paper depends on its venue. From the highest to the lowest quality the ranking
were i.e.: 4) Journal Articles; 3) Conference proceedings; 2) Industry reports; 1) Others. The rank of
articles will be used in the next task to be weighting the research focus of articles.
Last task in second phase is Analyzing and Clustering the Articles. The articles will be clustered based
on the research focus of the articles. Research focus can have more than one focus, and those
keywords will be weighted based on paper’s rank. Lastly, this study will answer the research
questions by finding the contributions of the papers. Based on the solution/proposed system, the
article will be clustered also with the RQ contribution they have been offered. The research question
will be mixed up by the focus of the paper to obtain how the research question will be answered.
On the last phase, the result will be comprehensively reported as Systematic Literature Review. All
resources that have been used in this research will reported in detail. In this phase the conclusion
will be conducted based on findings and analysis result from articles.

Results and Discussion

Repositories Searches
Articles Repositories used in this study were: 1) ScienceDirect; 2) IEEE Xplore; 3) ACM Digital Library.
Searching on those repositories using search engine provided on those repositories. In ScienceDirect
the resulted searches can archived to zip file contain related article’s files. In IEEE Xplore and ACM
Digital Library, resulted searches can only be exported into a comma separated files that contain list
of related articles. Articles from ScienceDirect then move to next stage whilst articles from IEEE
Explore and ACM Digital Library should be searched for its particular article. The total of articles
resulted in this step were shown in Table 1.
Inclusion and Exclusions of Articles
After all the papers were collected, the title and abstract will be explored to determine if a paper
related to the study or not. Title will be seen for its words and context whether it contain the
research theme or not. For Abstract of article, it will be explored in the proposed part of it. The
result of this step shown in Figure 5 below. Total articles have been reviewed in this task are 110
articles consists of 49 IEEE articles, 32 ScienceDirect articles and 29 ACM Digital Library articles. IEEE
searches resulted the most of articles and almost a half of it related (49% related). The most related
articles are articles from ScienceDirect, more than a half were related to the research (59% related).
The less related articles are from ACM Digital Library, 17% from 29 articles were related.
 30
25
25 24 24

20 19

15 13

10
5
5

0
IEEE ScienceDirect ACM Digital Library

Related No Related

Figure 5. Related Papers

Article Ranking
The rank of the articles is based on the study design in [7], the article’s venue in which the articles
were published ranks as follow (from highest to lowest quality): 1) Journal Articles, 2) Confrence
Proceedings, 3) Industry reports, 4) Others (e.g. Web Article, Wikipedia, etc.). The rank will used to
weight the quality of articles. The articles will be grouped by its ranks, Table 4 shown the group of
articles and Group of Articles based on its qualities can be shown in Figure 6
Table 2. Group of Paper by Its Ranks

Type of Article Rank Papers

Journal Paper 4 [12] [2] [13] [14] [4] [15] [6] [16] [17] [5] [18] [19] [20] [21] [22] [23]
[24]
Conference Proceeding 3 [2] [3] [25] [11] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36]
[37] [38] [39] [40] [41] [42] [43] [44] [25] [11] [12] [45] [40] [46] [47]
Industry Reports 2 [48]
Others 1 -

Qualities
35
30
30
25
20
15 17
10
5
0 1
Journal Confrence Industry Others
0
Report

Figure 6. Group of Articles based on its Qualities

Clustering and Analysis
This task is the last task of Second Phase. By studying through the content of articles, research
focuses were filtered. An article can have more than one research focus. Research focuses were
major concept, main topic or additional methodology and framework to support the main issues on
a particular article. In a literature review-type article, research focus usually reside in Result or
Discussion section whilst on technical-type article the research focus can be exist in State of The Art
or Proposed Solution section. Research Focus cluster shown in Table 4 and Research Focuses
Distribution shown in Figure 8.
The other cluster resulted from this task is Article Cluster based on its RQ Contribution. RQ
Contribution searched mainly in the result or proposed solution section. Like the other cluster,
Articles mapped its result whether it contribute to one or more RQ or none. RQ Contribution shown
in Table 3 below and RQ Contribution Distribution visualize in Figure 7.
Table 3. RQ Contribution

Ref RQ1 RQ2 RQ3 RQ4

[12] - - ✓ -
[2] - ✓ - ✓
[13] - - ✓ ✓
[14] ✓ - - ✓
[4] - - - ✓
[15] ✓ ✓ - -
[6] ✓ ✓ ✓ ✓
[16] ✓
[17] - - - -
[5] ✓
[18] - ✓ - -
[19] ✓ ✓ ✓
[20] - - - -
[21] ✓
[22] - - - -
[23] - - - -
[24] - - - -
[3] ✓ ✓ ✓
[25] ✓ ✓ ✓
[11] ✓ ✓
[26] - - - -
[27] ✓
[28] ✓
[29] ✓ ✓ ✓
[30] ✓ ✓
[31]
[32] ✓ ✓
[33] ✓ ✓ ✓
[3] ✓ ✓ ✓
[34] ✓ ✓
[35] ✓ ✓
[36] ✓
[37] ✓ ✓
[39] ✓ ✓
[38] ✓ ✓
[41] ✓ ✓ ✓
 [42] - - ✓ -
[43] - - ✓ -
[44] - - - -
[25] - ✓ ✓ ✓
[11] ✓ - ✓ -
[12] ✓ - ✓ -
[2] ✓ - ✓ -
[45] - ✓ - ✓
[5] ✓ ✓ ✓ -
[46] ✓ - ✓ -
[47] - - ✓ -
[48] - - - -

RQ Contributi ons
RQ4

RQ3

RQ2

RQ1

0 5 10 15 20 25 30

Figure 7. RQ Contributions Distributions

In this tasks
 Table 4. Article Cluster based on Research Focus

Ref. Rank SQ IaaS SaaS SF OI CM DoT RoM CC LCM RM PM SDLC AM RA ITS SM

[12] 4 ✓ ✓ ✓ ✓ Cod
e
[2] 4 ✓ ✓ ✓ ✓ ✓ SQ Software Quality
[13] 4 ✓ ✓ ✓ ✓ IaaS Infrastructure as a Service
[14] 4 ✓ ✓ ✓ SaaS Software as a Service
[4] 4 ✓ ✓ ✓ ✓ ✓ SF Success Factor
[15] 4 ✓ ✓ ✓ ✓ ✓ ✓ OI Organization Impact
[6] 4 ✓ ✓ ✓ CM Change Management
[16] 4 ✓ ✓ ✓ ✓ ✓ DoT DevOps Tools
[17] 4 ✓ ✓ RoM Role of Management
[5] 4 ✓ ✓ ✓ ✓ ✓ CC Cloud Computing
[18] 4 ✓ ✓ ✓ ✓ LC Life Cycle Management
M
[19] 4 ✓ ✓ ✓ ✓ ✓ ✓ ✓ RM Risk Management
[20] 4 ✓ PM Project Management
[21] 4 ✓ ✓ SDL SDLC
C
[22] 4 ✓ ✓ ✓ AM Agile Methodology
[23] 4 ✓ RA Risk Analysing
[24] 4 ✓ ✓ ITS IT Security
[3] 3 ✓ ✓ ✓ SM Statistical Methods
[25] 3 ✓ ✓ ✓ ✓ ✓
[11] 3 ✓ ✓ ✓
[26] 3
[27] 3 ✓ ✓
[28] 3 ✓ ✓ ✓ ✓
[29] 3 ✓ ✓ ✓
[30] 3 ✓ ✓ ✓ ✓
[31] 3
[32] 3 ✓ ✓ ✓ ✓
[33] 3 ✓ ✓ ✓ ✓
[3] 3 ✓ ✓ ✓
[34] 3 ✓ ✓ ✓
[35] 3 ✓ ✓ ✓
[36] 3 ✓ ✓ ✓
[37] 3 ✓ ✓ ✓
[39] 3 ✓ ✓
[38] 3 ✓ ✓ ✓ ✓
[41] 3 ✓ ✓ ✓ ✓ ✓
[42] 3 ✓ ✓ ✓
[43] 3 ✓ ✓ ✓
[44] 3 ✓ ✓ ✓
[25] 3 ✓ ✓ ✓
[11] 3 ✓ ✓ ✓ ✓
[12] 3 ✓ ✓ ✓ ✓ ✓
[2] 3 ✓ ✓ ✓ ✓ ✓
[45] 3 ✓ ✓ ✓ ✓ ✓ ✓
[5] 3 ✓ ✓ ✓
[46] 3 ✓ ✓ ✓ ✓ ✓
[47] 3 ✓ ✓ ✓ ✓ ✓ ✓
[48] 2 ✓ ✓ ✓
 Focus

67 64
58
54
45 43
Freq. 36
26 28
20 21 22 23
16 17 18
12 11 14 13 14 1212
10 7 8 8 7 7
4 4 4
SQ IaaS 3 SF OI CM DOT RoM CC LCM RM PM AM RA 3
ITS SM
SaaS SDLC
Unweighted 10 4 3 20 16 14 17 7 8 8 7 13 4 18 7 4 3
Weighted 36 12 11 67 54 45 58 21 26 28 22 43 14 64 23 12 12

Figure 8. Research Focuses Distribution

Research Focuses
Software Quality

IaaS
Current industry trend of DevOps leverages the improvement cloud technology especially
Infrastructure as a Service (IaaS). IaaS is a platform that support DevOps in their daily operations. As
example, CloudWave is a IaaS platform that feeds runtime performance and monitoring insight back
to the DevOps Engineers to enable identification and refactoring of problematic code sections [28].
To enable continuous delivery, [42] proposed a framework for automated provisioning of application
by utilizing Configuration Management, in this study, DevOps team can be deploy the application as
they push the code to the source repository.
SaaS

DevOps Success Factor

DevOps Succes factor is a something important to implement DevOps in organization.
Especially for a organization that want to adopt the DevOps method to their environment,
organization need some key success to design good conection between Development and
Operation. [40] design systemic based on system engineering and cybernetics principlles, this desgin
build to re-inforce and establish conection on Development and operation. This paper tracing the
DevOps key strategic succes factors. this approach enables program managers to ensure that the
program’s progress is effectively tracked to success. [12] review about try to controlling risk and
dealing with dynamics of the environment of program and projects. Find the result pays more
attention on soft controls if compared with traditional method. [15] [16] reveal that there is a few
key to measure related factor to Continuous deployment (CD) and support tools for different
development test on CD.
[5] V. Gupta and teams reviews about measure and asess the DevOps attributes using
statistical analysis and two-way asessment. There are 18 attributes define and assess 18 attributes to
identify key independent attributes and measure them to de-termine the maturity of DevOps
implementation. This framework contributes significantly to the field of DevOps by enabling prac-
titioners to conduct the detailed assessment and measurement of DevOps attributes to determine
the maturity of DevOps implementation to achieve higher quality.[29] M. Shahin This study aims at
empirically investigating and classifying the factors that may impact on adopting and implementing
CD practice on organization. And with the analysis M. Shahin finding 11 factors that limit or
demotivate software organizations to push changes automatically and continuously to production.
Impact on Organization
Adopting DevOps to organization will make some changes or impact for the organization
such as how fast the software delivery on organiztion after they are adopt DevOps method. [12]
define the organization impact of DevOps method is working with such approaches as DevOps
require a continuous link and communication between all the relevant stakeholders which does
allow for improved detection of early warning signals when it relates to risks. [33] share about how
to make efficient and effective organization with implementation autonomic computing.

Project Management
Project Management is the practice of initiating, planning, executing, controlling, and closing
the work of a team to achieve specific goals and meet specific success criteria at the specified time.
[22] [34] reveal that project management at least have minimum specification for release the
product or feature which has minimum standard instrumentation, designand manage experiment
plans, link experiment results with a product roadmap, and manage a flexible business strategy.
In list project management there are several things to discuss related to DevOps research
focus such a change management, role of management, Life-cycle management and Risk
Management.

- Change Management
Change management is a systematic approach to dealing with the transition or
transformation of an organization's goals, processes or technologies. It includes methods
that redirect or redefine the use of resources, business process, budget allocations, or other
modes of operation that significantly change a company or organization. [49] provide
method if any changes in the management process are overlooked, this lack of changes can
cause inconsistencies between an application and its management, resulting in improper
system status. This method can recommend the necessary operation changes with high
accuracy and can reduce the possibility of operators overlooking them.

- Role of Management
Role management helps you manage authorization, which enables you to specify the
resources that users in your application are allowed to access.

- Life-cycle management & SDLC

Business management approach that can be used by all types of business (and other
organizations) in order to improve their sustainability performance. [25] P. Perera this paper
analysis and measure wheter there is a relationship between quality responsive to business
needs and agility with implementation of DevOps. Analyzing DevOps has a positive impact
on Quality, Responsiveness to Business Needs and Agility to New Technologies in Sri Lanka.

DevOps Tools & Cloud Computing

DevOps tools is a tools or application that supports the process of the devops method such
as continuous delivery (CD). [27] create validation automated deploying with regression test, One of
the challenges of automated deploying of new releases to production environment, is software
reliability, that in case it is breached, the beneficiaries would suffer considerable losses. Regression
test put on CD steps and validate the application last version as black box and prevent writing test
unit. [42] [37] create and implement the architecture of clouds on DevOps environment.

Risk Management & Risk Analyzing

Risk management is the identification, evaluation, and prioritization of risks followed by
coordinated and economical application of resources to minimize, monitor, and control the
probability or impact of unfortunate events. [19] this study focuses on identifying challenges when
adopting the CD process as well as strategies tomitigate these challenges. The study incorporates
two main per-spectives to challenges when adopting to the CD process, technicaland social adoption
challenges. [32] this paper research approach to support developers in their decision-making by
incorporating runtime information in source code and provide live feedback in IDEs by predicting the
impact of code changes.
Agile Methodology
The Agile Method is a particular approach to project management that is utilized in software
development. This method assists teams in responding to the unpredictability of constructing
software.

IT Security
IT Security is the protection of computer systems from theft or damage to their hardware,
software or electronic data, as well as from disruption or misdirection of the services they provide.
[41] V. Mohan and L. Ben Othmane review about security as a part of DevOps. This paper try to
prove that security is a important pert for DevOps, and add the Security to DevOps as SecDevOps.
[23] [47] [48] reveal about strengthen security on the organzation especially for DevOps
environment.

Statistical Study
Statistical study is to address these difficulties through learning from the correlation
between resources and operations’ activities. [17][5][24] paper that use statistical study to
measure influence DevOps.

In this paper Systematic Literature Review, there are a few examples of reviews on DevOps,
as [5] V. Gupta and teams review about how to assess and measure the DevOps attributes using
statistical analysis and Two-way assessment; in succes factor [40] S. W. Hussaini approach the
method to re-inforce collaboration Development and Operations functions; Measurement and
Sharing are important factors to be considered when implement DevOps on the organization and
recommended to implement and practice DevOps [24] P. Perera; [29] M. Shahin review about succes
factor and factor that can interfere organization to implement DevOps; [40] V. Mohan and L. Ben
Othmane review SecDevOps (Security Development Operations) as the important part for DevOps.

RQ Contributions

Conclusions
1. Impact DevOps to Organization.
2. How far Organization in this time ready for adoption DevOps
3. Find success factor for DevOps Initiation
4. Design change management for DevOps Implementation
References
[1] F. Erich, C. Amrit, and M. Daneva, “Report: DevOps Literature Review,” Univ. Twente, no.
October, p. 27, 2014.
[2] D. Eaves, “Modern DevOps: Optimizing software development through effective system
interactions,” Handb. Polym. Foam., pp. 1–8, 2004.
[3] M. B. Kamuto and J. J. Langerman, “Factors inhibiting the adoption of DevOps in large
organisations: South African context,” RTEICT 2017 - 2nd IEEE Int. Conf. Recent Trends
Electron. Inf. Commun. Technol. Proc., vol. 2018–Janua, pp. 48–51, 2018.
[4] L. Chen, “Continuous Delivery: Overcoming adoption challenges,” J. Syst. Softw., vol. 128, pp.
72–86, 2017.
[5] V. Gupta, P. K. Kapur, and D. Kumar, “Modeling and measuring attributes influencing DevOps
implementation in an enterprise using structural equation modeling,” Inf. Softw. Technol.,
vol. 92, pp. 75–91, 2017.
[6] B. Fitzgerald and K.-J. Stol, “Continuous software engineering: A roadmap and agenda,” J.
Syst. Softw., vol. 123, pp. 176–189, Jan. 2017.
[7] B. Kitchenham, “Procedures for performing systematic reviews,” Keele, UK, Keele Univ., vol.
33, no. TR/SE-0401, p. 28, 2004.
[8] K. Schwalbe, “Information Technology Project Management,” p. 643, 2016.
[9] E. J. Brown and J. William A. Yerberry, The Effective CIO, How to Achieve Outstanding Success
through Strategic Alignment, Financial Management and IT Governance. Auerbach
Publications Taylor & Francis Group, 2009.
[10] E. M. Schön, J. Thomaschewski, and M. J. Escalona, “Agile Requirements Engineering: A
systematic literature review,” Comput. Stand. Interfaces, vol. 49, pp. 79–91, 2017.
[11] M. Rajkumar, A. K. Pole, V. S. Adige, and P. Mahanta, “DevOps culture and its impact on cloud
delivery and software development,” Proc. - 2016 Int. Conf. Adv. Comput. Commun. Autom.
ICACCA 2016, 2016.
[12] R. Bierwolf, P. Frijns, and P. Van Kemenade, “Project management in a dynamic environment:
Balancing stakeholders,” Proc. 2017 IEEE Eur. Technol. Eng. Manag. Summit, E-TEMS 2017,
vol. 2017–Janua, pp. 1–6, 2017.
[13] M. Makki, D. Van Landuyt, B. Lagaisse, and W. Joosen, “A comparative study of workflow
customization strategies: Quality implications for multi-tenant SaaS,” J. Syst. Softw., vol. 144,
no. February, pp. 423–438, 2018.
[14] T. Mikkonen, C. Lassenius, T. Männistö, M. Oivo, and J. Järvinen, “Continuous and
collaborative technology transfer: Software engineering research with real-time industry
impact,” Inf. Softw. Technol., vol. 95, no. November 2017, pp. 34–45, 2018.
[15] P. Rodríguez et al., “Continuous deployment of software intensive products and services: A
systematic mapping study,” J. Syst. Softw., vol. 123, pp. 263–291, 2017.
[16] S. Mäkinen et al., “Improving the delivery cycle: A multiple-case study of the toolchains in
Finnish software intensive enterprises,” Inf. Softw. Technol., vol. 80, pp. 1339–1351, 2016.
[17] M. Farshchi, J. G. Schneider, I. Weber, and J. Grundy, “Metric selection and anomaly
detection for cloud operations using log and metric correlation analysis,” J. Syst. Softw., vol.
137, pp. 531–549, 2018.
[18] J. Dunne and D. Malone, “Obscured by the cloud: A resource allocation framework to model
cloud outage events,” J. Syst. Softw., vol. 131, pp. 218–229, 2017.
[19] G. G. Claps, R. Berntsson Svensson, and A. Aurum, “On the journey to continuous
deployment: Technical and social challenges along the way,” Inf. Softw. Technol., vol. 57, no.
1, pp. 21–31, 2015.
[20] R. Vallon, B. J. da Silva Estácio, R. Prikladnicki, and T. Grechenig, “Systematic literature review
on agile practices in global software development,” Inf. Softw. Technol., vol. 96, no.
December 2017, pp. 161–180, 2018.
[21] T. Karvonen, W. Behutiye, M. Oivo, and P. Kuvaja, “Systematic literature review on the
impacts of agile release engineering practices,” Inf. Softw. Technol., vol. 86, pp. 87–100, 2017.
[22] F. Fagerholm, A. Sanchez Guinea, H. Mäenpää, and J. Münch, “The RIGHT model for
Continuous Experimentation,” J. Syst. Softw., vol. 123, pp. 292–305, 2017.
[23] K. Tuma, G. Calikli, and R. Scandariato, “Threat analysis of software systems: A systematic
literature review,” J. Syst. Softw., vol. 144, no. May, pp. 275–294, 2018.
[24] G. Schermann, J. Cito, P. Leitner, U. Zdun, and H. C. Gall, “We’re doing it live: A multi-method
empirical study on continuous experimentation,” Inf. Softw. Technol., vol. 99, no. March
2017, pp. 41–57, 2018.
[25] P. Perera, M. Bandara, and I. Perera, “Evaluating the impact of DevOps practice in Sri Lankan
software development organizations,” 16th Int. Conf. Adv. ICT Emerg. Reg. ICTer 2016 - Conf.
Proc., no. March 2018, pp. 281–287, 2017.
[26] J. Bruel, “Reconciling Requirements and Continuous Integration in an Agile Context,” pp. 508–
509, 2018.
[27] M. J. Kargar and A. Hanifizade, “Automation of regression test in microservice architecture,”
2018 4th Int. Conf. Web Res. ICWR 2018, pp. 133–137, 2018.
[28] E. Salant, P. Leitner, K. Wallbom, and J. Ahtes, “A framework for a cost-efficient cloud
ecosystem,” eChallenges e-2015 Conf. Proc., 2016.
[29] M. Shahin, M. A. Babar, M. Zahedi, and L. Zhu, “Beyond Continuous Delivery: An Empirical
Investigation of Continuous Deployment Challenges,” Int. Symp. Empir. Softw. Eng. Meas.,
vol. 2017–Novem, pp. 111–120, 2017.
[30] T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck, and M. Stumm, “Continuous
deployment at Facebook and OANDA,” Proc. 38th Int. Conf. Softw. Eng. Companion - ICSE ’16,
pp. 21–30, 2016.
[31] A. van Deursen, “Software engineering without borders,” 2017 32nd IEEE/ACM Int. Conf.
Autom. Softw. Eng., p. 3, 2017.
[32] J. Cito, “Developer targeted analytics: supporting software development decisions with
runtime information,” Proc. 31st IEEE/ACM Int. Conf. Autom. Softw. Eng. - ASE 2016, pp. 892–
895, 2016.
[33] H. Li, T.-H. P. Chen, A. E. Hassan, M. Nasser, and P. Flora, “Adopting autonomic computing
capabilities in existing large-scale systems: An industrial experience report,” Proc. - Int. Conf.
Softw. Eng., pp. 1–10, 2018.
[34] M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in
Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,”
Proc. - 2015 IEEE Int. Conf. Cloud Comput. Emerg. Mark. CCEM 2015, pp. 85–89, 2016.
[35] S. Palihawadana, C. H. Wijeweera, M. G. T. N. Sanjitha, V. K. Liyanage, I. Perera, and D. A.
Meedeniya, “Tool support for traceability management of software artefacts with DevOps
practices,” 3rd Int. Moratuwa Eng. Res. Conf. MERCon 2017, pp. 129–134, 2017.
[36] G. Rong, H. Zhang, and D. Shao, “CMMI guided process improvement for DevOps projects,”
Proc. Int. Work. Softw. Syst. Process - ICSSP ’16, pp. 76–85, 2016.
[37] D. Bruneo et al., “CloudWave: Where adaptive cloud management meets DevOps,” Proc. -
Int. Symp. Comput. Commun., vol. Workshops, pp. 1–6, 2014.
[38] S. W. Hussaini, “Strengthening harmonization of Development ( Dev ) and Operations ( Ops )
silos in IT environment through Systems approach .,” pp. 178–183, 2014.
[39] W. Hussain, T. Clear, and S. MacDonell, “Emerging trends for global DevOps: A New Zealand
perspective,” Proc. - 2017 IEEE 12th Int. Conf. Glob. Softw. Eng. ICGSE 2017, pp. 21–30, 2017.
[40] S. W. Hussaini, “A Systemic Approach to Re-inforce Development and Operations Functions in
Delivering an Organizational Program,” Procedia Comput. Sci., vol. 61, pp. 261–266, 2015.
[41] V. Mohan and L. Ben Othmane, “SecDevOps: Is it a marketing buzzword? Mapping research
on security in DevOps,” Proc. - 2016 11th Int. Conf. Availability, Reliab. Secur. ARES 2016, pp.
542–547, 2016.
[42] N. K. Singh, S. Thakur, H. Chaurasiya, and H. Nagdev, “Automated provisioning of application
in IAAS cloud using Ansible configuration management,” Proc. 2015 1st Int. Conf. Next Gener.
Comput. Technol. NGCT 2015, no. September, pp. 81–85, 2016.
[43] N. M. Devadiga, “Software Engineering Education: Converging with the Startup Industry,”
Proc. - 30th IEEE Conf. Softw. Eng. Educ. Training, CSEE T 2017, vol. 2017–Janua, pp. 192–196,
2017.
[44] A. Furfaro, T. Gallo, A. Garro, D. Saccà, and A. Tundis, “ResDevOps: A Software Engineering
Framework for Achieving Long-Lasting Complex Systems,” Proc. - 2016 IEEE 24th Int. Requir.
Eng. Conf. RE 2016, pp. 246–255, 2016.
[45] S. Kitajima, S. Kikuchi, and Y. Matsumotoy, “Operation changes recommendation method
using histories of operation changes in cloud computing environment,” Proc. - IEEE 7th Int.
Conf. Cloud Comput. Technol. Sci. CloudCom 2015, pp. 322–330, 2016.
[46] J. Domaschka, F. Griesinger, D. Baur, and A. Rossini, “Beyond Mere Application Structure
Thoughts on the Future of Cloud Orchestration Tools,” Procedia Comput. Sci., vol. 68, pp.
151–162, 2015.
[47] G. Casale et al., “Current and Future Challenges of Software Engineering for Services and
Applications,” Procedia Comput. Sci., vol. 97, pp. 34–42, 2016.
[48] J. Topper and T. S. Factory, “Compliance is not security,” Comput. Fraud Secur. Bull., vol.
2018, no. 3, pp. 5–8.
[49] S. Kitajima, S. Kikuchi, and Y. Matsumoto, “Operation Changes Recommendation Method
using Histories of Operation Changes in Cloud Computing Environment,” 2015.