RESILJAr)', ITIL®. PRINCE2® MSP®. MoP® luKi ~iOV® Iln.

:Registered Trade ~itlrks of AXEL OS in the Ulliu:d Kingdom end other countries
COOIT® is 3 registered IrndcJllllrk oflhc Inforneuion Syt'tcnls AIKtit tII~ Comml Associ:llioll (IS;\CA) and Ihe rr Governance lnsritutc (ITGI)
TOGAFlM and IT4rrtM arc ImdC!mJrks onne Open G«)UP
SIAl\i® is u registered trademark 01"EXIN

o 2018of CFN Conwlt unless otha"";5e 511l~1Id

Agenda

1. Why DevOps?

2. DevOps principles

3. DevOps concepts

4. DevOps practices

5. DevOps people

6. DevOps controls

7. DevOps training and further reading

8. Where do you start with DevOps?

eFN
02018 • "1' ••
Why OevOps?

To reduce deployment lead time to minutes!

) The business demands faster and continuous delivery.
o Most organisations are not able to deploy production
changes in minutes or hours, instead requiring weeks or
months.
o Opposing goals between
Development and Operations:
Conflict between agile
development (urgent projects)
and stable operation
(keep it running, don't mess
with the environment)

eFN CA... ,,,ie

02018 • "1' ••
Typical problems between Dev and Ops

J Organizational silos _ .....

J Different mindsets
o Different tools
'.
J Different environments
o Product back-logs
J Blame game
J Disintegrated processes
Devetopment wutl of confusion Operattens
.:J Poor feedback loops

Leads to a downward spiral: Everybody gets a little

busier, work takes a little more time, communications
become a little slower, and work queues get a little
longer. Work requires more communication,
coordination, and approvals. CFr-.
02018 • "1' u.
What is DevOps?

Teardawn the wall between Development and Operations:

Continuous Delivery: Continuous Integration, Testing and Deployment

1III1,\·(nlfiQI1:
R(rtlt}lIt' 81Ulllt,,(JlII'1

"Small teams independently implement their features, validate their correctness

in production-like environments, and have their code deployed into production
quickly, safely and securely."
The DetIOps HO(ldbook. by Gene Kim, lel Humble, Poulck Debols ond IOhft Wi/Us, 1016

02018 . .•.
..,
What is DevOps?

"Continuous Delivery is about putting the release schedule in the hands of

the business, not in the hands of IT. Implementing Continuous Delivery
means making sure your software is always production ready throughout its
entire lifecycle - that any build could potentially be released to users at the
touch of a button using a fully automated process in a matter of seconds or
minutes.

This in turn relies on comprehensive automation of the build, test and

deployment process, and excellent collaboration between everyone involved
In delivery - developers, testers, DBAs, systems administrators, users, and
the business. In the world of Continuous Delivery, developers aren't done
with a feature when they hand some code over to testers, or when the
feature is "QA passed". They are done when it is working in production. That
means no more testing or deployment phases, even within a sprint (if you're
using Serum)."

ContfnlJOUSOellvto/..Rellobfe So/cW()tt ReleoSl:S ("'ough Build, Test, ond Depfoyment AVfontolfon, by Jel Hvmhle
and David Forley. 2010
CF!'
02018 • "'1' u.
What is DevOps?

eFN Cu,,<,,11
02018 • ""l' HI
A brief history of DevOps

, DevOps relies on old and proven concepts such as lean, it service management, agile
devetopment, theory of constraints, resilience engineering, learning organisations, etc,
J At the 2008 Agile conference in Toronto, Canada, Patrick Debois and Andrew Schafer held a
"birds of a feather" session on applying Agile principles to infrastructure as opposed to
application code.
) Later, at the 2009 Velocity conference, John Allspaw and Paul Hammond gave the semlnat
"10 Deploys per Day: Dev and Ops Cooperation at Flickr" presentation.
, Patrick Debois was not there, but was so exited by Allspaw and Hammond's idea that he
created the first DevOps Days in Ghent, Belgium in 2009. There the term "OevOps" was
coined.
J In 2010 Jez Humble and David Farley in their book "Continuous Delivery", extended the
concept 10 continuous delivery, which defined the role of a "deployment pipeline" to ensure
that code and infrastructure are always in a deployable stale, and lhat all code checked in 10
trunk can be safely deployed into production.
, In 2013 Gene Kim et. al. published their novel "The Phoenix Project: A Novel AboullT,
DevOps, and Helping Your Business Win" that made DevOps commonly known in the
Industry.
J In 2016 Gene Kim, Jez Humble, Patrick Debois and John Willis published "The OevOps
Handbook"

02018
eFr-. • "1' u.
Principles, Concepts, Practices and People

Flow 1. Product 1. Continuous 1. Customer centric

2. Feedback 2. Value stream integration 2. End-to-end
3. Continual 3. Loosely-coupled 2. Fastand reliable responsibility
learning and architecture automated 3. Collaboration
improvement testing
4. Autonomous 4. Learningand
teams 3. Continuous and improvement
automated
deployment / 5. Experimentation
and risk taking
provisioning
4. Measurement
and feedback

02018
eFr-. '1' HI
The three OevOps Principles
1. FLOW

Enable fast left-to-right flow of work from

Development to Operations to the customer

~ Make work visible using visual boards

~ limit work in process (WIP) Dev Ops
J Reduce batch sizes
J Reduce the number of handovers
J Continually identify and elevate constraints
J Eliminate hardships and waste in the value
stream

02018 ,. .
<:.. 11,-1111
The three OevOps Principles
2. FEEDBACK

Enable fast and constant flow of feedback from

right to left at all value stream stages

> Establish fast feedback loops at every step

j
of the process
Establish pervasive production telemetry
ensuring that problems are detected and
corrected as they occur
Dev
...
......._ _-_ ..../
Ops

j Keep pushing quality closer to the source

j Enable optimising for downstream work
centers

CFi\ (:.. I1..lIh

02()18 l' ..
The three OevOps Principles
3. CONTINUAL LEARNING & IMPROVEMENT

Enable a high-trust, experimenting and risk-

taking culture as well as organisational learning,
both from successes and failures

r ---.
> Enabling organisational learning Dev ~ Ops
j Institutionalise the improvement of daily ,..,__ ./
work
j Transform local discoveries into global
improvements
j Inject resilience patterns into daily work
j Encourage leaders to reinforce a learning
culture
J Experiment, fail fast - If it hurts, do it more
often
CF[\.
02()18
Principles, Concepts, Practices and People

Flow 1. Product 1. Continuous 1. Customer centric

2. Feedback 2. Value stream integration 2. End-to-end
3. Continual 3. Loosely-coupled 2. Fastand reliable responsibility
learning and architecture automated 3. Collaboration
improvement testing
4. Autonomous 4. Learningand
teams 3. Continuous and improvement
automated
deployment / 5. Experimentation
and risk taking
provisioning
4. Measurement
and feedback

02018
eFr-. '1' HI
Product

Our application is our product. Strive towards detivering a minimum viable

product (MVP), the smallest amount of functionality having value to the
customer, as fast and reliably as it can.

The minimum viable product (MVP)

reflects the end-product in a minimal
functional form. It is used to test new
ideas and verify whether the hypothesis
are correct.

LikJ! this!
"The minimum viable product (MVP) is
that product which has just those features
and no more that allows you to ship a
product that early adopters see and, at ~
least some of whom resonate with, pay t
you money for. and start to give you IIIU$frotlon; Hel'lrfk Knlbefg
feedback on"
Tho L08n St8rtUp. by Eric Rio s, 2011
CFr-.
02018 • "'1' u.
 Product
Theme

Epic Epic

Feature Feature Feature Feature

Time
Sprint 1 Story Story
MVP
Story

Sprint 2 Story Story

Sprint 3 Story Story

Backlog Story Story Story

02018
CF" . ..,.. ~
Product
J Create with the End in Mind: Understanding the real needs of customers .
.J Before we build a feature, we should rigorously ask ourselves, "Should
we build it, and why?'
J We should then perform the cheapest and fastest experiments possible
to validate through user research whether the intended feature will
actually achieve the desired outcomes.
J Think about each feature as a hypothesis and use production releases as
experiments with real users to prove or disprove that hypothesis.
J We can use techniques such as hypothesis-driven development,
customer acquisition funnels, and NB testing.

23 EUR
27%
I'lver1ge
con ...erslon
order sue
SO% of vlsl10rs see
variation A

1-
27 EUR
13%
.w~ra&e
conversion
order sue
50% of visitors see
variation 9 Vatlatlon 8 eFN
02018 • "1'_.
Value stream

The series of events that take a feature from code commit through to
production.

Commit Acceptan· Explorato-

stage

(Build.
package,
contlnuou~
intcgration.
automated
ce stage
(Deployment
. automated
acceptance
lest~)
ry testing

(Manuallv
detect
defects not
caugru by
eutemated
..
unit tests) tests)

Th~deplayment prp~Unt.l,omConrinuous Delivery. by J~lHumblt ond David Fodt"/. 2010

Use production-like environments at every stage of the value stream

Enable on-demand creation of Dev, Test, and Production environments.
Environments must be created in an automated manner, ideally on demand
from scripts and configuration information stored in version control, and
entirely self-serviced, without any manual work required from Operations

02018
eFr-. U)l151111
'1' .~
Value stream mapping

1- -I
I

Commit Commit Acceptance

Integration Unit test Deployment
-c~o::
00~{
err- err= crr=
CIO= c/o= I
c/o=
%c/A= I %CIA= %c/A=
Uptime= Uptime: Uptime=
Leedume
(Non·value
O.Sd O.Sd add.dUma)

~ ~ { ~ ...._4_hou_rs-l 2 hours 30 min ~ssln9

1- ..1 lime (Value
added time)

CIT: Cycle TIme; C/O: ChangeoverTIme;fJ4CjA: Percent complete and accurate; I: Inventory

eFN
02018 .. "1'_.
Loosely-coupled architecture

DevOps architecture principles

~ Componentization via SelVices: It is independently deployable, ctoud-ready. and scalable.
_) Organized Around Business Capabilities: Organizations are organized around business
capabilities (Micro Service Architectures· MSAs)
J Products not Projects: The characteristic aopreclates 'You build it, you run it: This involves
taking full responsibility.
_) Smart Endpoints and Dumb Pipes: These are simple intertaces having no logic in between.
such as the Enterprise Service Bus.
~ Decentralized Governance: There is no standardization on a single technology platform.
_) Decentralized Data Management: Transactions help with consistency but imposes temporal
coupling.
_) Infrastructure Automation: Automated tests and automated deployments.
J Design for Failure Tolerance: Any service call can fail due to unavailability of the supplier.
Therefore, the client has to respond to this as gracefully as possible, detect the failures
quickly, and restore the service.
, Evolutionary Design: The design supports independent replacement and upgradeability.

CFr-.
02018 • "'1' u.
Loosely-coupled architecture

From monolith to microservices

Monolithic architecture Microservices architecture

02018
CFr-.
. ..,...
U)l151111
Loosely-coupled architecture
DevOps architecture archetypes
p(OS
Monolithic v1
(All functjonality in one
applttatJon)
Monolithic ..,2
(sets of monolithic tiers: "'front
end presemetlen ....-appliQllrOn
server", "detebese laver'"
Mi(lroservice
(Modular-Independent, Sr.lph
relatklnshlp YS.uers, Isolated
persistence)
From the Monolith to Mfcfoservfces, by Randy Shoup, 2015
Simple at first
low Imer-prccesstatenctes
Stnsle Ciode-base. one dcpJoymenl
Resource-effklent at small.scale
Slmple at fim
Join qcertes arc C:3Sy
SIngleschemil deployment
Resource-effKient at small scale
Each unit Is simple
Independent scaling and periormance
Independent testing and depJoyment
Can optimaliv tune performance
Icachlng, replication,. etCl.)
02018
Cons
C!oordlnatlon overhead increases as
tearngrcws
unit Poor ('nforccmont of modularity
Poo'~l1na
AIl-or-nolhTng deploy (downtime
failures)
long build times
Tendency (or Incrt!ased coupling over
lime
Poor scal1naand redundancy (illI or
nothing, vertical only)
Difficult to tune property
AII·or·nothlng schema m.,nag(!m('nl
Many coopt'ratlng units
Many-sm01IIrepositories
Requires more sophlstrQlly tooling ~nd
dependency management
Networ$; latencies
eFN
• "'1' UI
Loosely-coupled architecture

Good practices
o Automate building and configuring the environment using virtualized environments.
environment creation processes that start from "bare metal". "infrastructure as code"
configuration managemenllools. automated operating system configuralion tools.
assembting an environmenl from a set of virtual images or containers. spinning up new
environments In public cloud etc.
) Embrace 'everything as code' concepts. In Ihe Iraditional operations space, more and more
components can be defined wilh code. such as soflware-defined networks, Define more and
more artefacts with code,
, Creale our single repository of truth for Ihe entire system, Pul all application source files and
configurations as well as all produclion artefacts in version control. Version control is for
everyone in our value stream, including QA, Operations, Infosec, as well as developers
J Make Infrastruclure easier to rebuild than 10repair. When we can quickly rebuild and re-
create our applications and environments on demand, we can also quickly rebuild them
instead of repairing them when things go wrong,
) Design a loosely-coupled architeclure with well-defined Interfaces that enforce how modules
connect with each other 10promote productivity and safely,

02018
eFr-. • ""l' u.
Loosely-coupled architecture

Loosely-coupled architecture based on Microsoft cloud reference architecture

On premises Infrastructure Platform Software

Applications
Data Data
Runtime Runtime
Middleware Middleware

Virtualization
Servers
Storage
Networking
Host Build Consume

eFr-. UII1~111c
02018 .. "'l'.'
Autonomous teams

"A team is a small number of people with complementary skills who are committed to a common
purpose. set of performance goals. and approach for which they hold themselves mutually
responsible:
The Wisdom of Teems, by Kotzenbac.h & Smith. 1993

Cross-functional DevOps autonomous teams:

, consist of representatives from al/ disciplines fully accountable for developing and deploying
an IT service
J are fully empowered and self-sufficient to design. build. test. deploy. and run Ihe software
J operate autonomously and work independenlly from one another to deliver a continuous
stream of software change
J have end-to-end responsibility. There is no handover or transfer of responsibility and
accountability
) possess all the necessary expertise to take on the end-to-end responsibility
, needs team members with T-shaped profile and complementary skills
J provide support to products until end-of-life.
J are kept stable so they can keep iterating and improving and embed effective working hablts.

eFr-.
02018 • "'1' u.
Autonomous teams

Good practice
J Think values over culture
J Share stories and experiences
J What kinds of problems are we trying to solve?
) Are we solving the right problems?
J Does our team have the necessary knowledge and experience to acknowledge
the problem and to understand the repercussions that their potential solutions
might have?

Product owner

Developer ~
Operations engineer
ISys admin, mldcleware,
DBA. .. c.,
!
In(oSee
Release manager
eFr-.
02018 • ""l' u.
Autonomous teams

Allow small product teams to work safely and architecturally decoupled from the work
of other teams who use self-service platforms that leverage the collective experience
of Operations and Information Security

} Productteam'

- - - - - - - - - - (A.PlISeIfS(!(Vlce - - - - - - - - - - -

PaaS /r- W. .\,

Platform teams
laaS j
~ \!)!l~

02018
eFr-. .. ..,...
(.AU1~111c
Autonomous teams

Oecoupling of Product and Platform teams:

J Interfaces between Product teams and Platform teams are clearly defined
through Application Programming Interfaces (APls).
J The Platform teams offer a rich set of standardized self-service capabilities to
Product teams, such as logging, monitoring, deployment, backup, recovery and
many more. Such a set of capabilities/services allows Product teams to
completely manage their (software) products.
J The Product teams are the customers of the Platform teams. They use the
products of the Platform teams. These products can be used as fully automated
self services.
) The Platform teams are responsible for the qualities of their platform products,
such as availability and performance. The Product teams are responsible for the
qualities of their (application) products, such as availability and performance.
) The system qualities of the platform products can be monitored and managed
independently from the availability of the application products, which use the
platform producls.

02018
eFr-. • "'1' u.
Autonomous teams

"Teams should only be as large as can be fed with two pizzas"

Conway's Law, states that "organizations which design systems ...are constrained to
produce designs which are copies of the communication structures of these
organizations .... The larger an organization is, the less flexibility it has and the more
pronounced the phenomenon."
Source: Conway. Melvin E., 1968

Market-oriented organisations tend to be flat, composed of multiple, cross-functional

disciplines (e.g., marketing, engineering, etc.), which often lead to potential
redundancies across the organization.
This is how many organisations adopting OevOps operate. Market-oriented teams are
responsible not only for feature development, but also for testing, securing, deploying,
and supporting their service in production, from idea conception to retirement.
These teams are designed to be cross-functional and independent

CFr-.
02018 • "1' u.
Autonomous teams

Teams of teams
Scaling at Spotify with Tribes, Squads, Chapters & Guilds

...... _ ...... _
Source; hlrps://uCVOJ(./lfct.wordp(I!$$.com/ZOJ1/Jl/J1361790S·scofing-4gilt:·spoa/y·n.pdf

02018
(;Fi\
. ..,...
Autonomous teams

Measuring team performance at Spotify

Area I Squad 1 I Squad 2 I Squad 3 I Squad 4 I Squad 5 Legend

Product owner
O~ ®~ ®J)" 0..0..® NalGood

Agile coach ®~ @Ol I®. O¢1l 09_j 0 Ok

e
Influencing work
Easy to release
Process that fits team
O~ O~
O~ {II@
0..4IIlJi0l
O. ~~
e~
®J)~ CIIiJI~
O~
O~
®.
$<0
~ Im,..w,g

..
Good

&ellCly

A mission O<f;l{II~ O~ O~ ®. n ~ W....

Organization support ® .. e 0 0.. 0
Source: hrtps:/ /ucvox.jiles.wordpre5s.com/20J 2/J l/J 1361'190S-scoling·oglfe·sporify·l1.pdf

CFr-. UU151111
02018 • "'l'.'
Principles, Concepts, Practices and People

Flow 1. Product 1. Continuous 1. Customer centric

2. Feedback 2. Value stream integration 2. End-to-end
3. Continual 3. Loosely-coupled 2. Fastand reliable responsibility
learning and architecture automated 3. Collaboration
improvement testing
4. Autonomous 4. Learningand
teams 3. Continuous and improvement
automated
deployment / 5. Experimentation
and risk taking
provisioning
4. Measurement
and feedback

02018
eFr-. '1' HI
OevOps practices

1. COntinuou-s 3. Continuous and automated

integration 2. Fast and reliable automated testing deployment and provisioning

•
4. Measurement and feedback
: Approval
The deployment pIpeline, from COIIt/nuou$CJelllll!fYJby If!l Humble ond DCtvidForley. ZOJO

CF~
02018 • "'l'.'
Continuous integration

Continuous Integration (CI) Is the practice, in software engineertng, of merging all developer
working copies to a shared mainline several times a day,
J Once a team member commits a number of code changes into Software Configuralion
Management, the changes can be merged automatically, analysed, compiled, unit lested,
and assembled automatically.
) The automated build process can create a new deployment package and publish it to a trunk.
In this manner, a continuous now from code commit 10validated deployment package can be
implemenled.
) The automated build process is important for a fail·fast strategy. For example, if there are
code merge confltcts, the build should fail so the team members can fix the conflicts.

Applkation ccrnpcnem
I> U.e. miCtoscrvlce)

Development SCM commit Sulld Pipeline Test

CFf\
02018 • "1' u.
Continuous integration

Good practice
J Adopt trunk-based development practices where all developers check in
their code to trunk at least once per day.
) Frequent code commits to trunk means that all automated tests can be
run on the software system as a whole and alerts received when a
change breaks some other part of the application or interferes with the
work of another developer.
J The discipline of daily code commits forces the developers to break the
work down into smaller chunks while still keeping trunk in a working.
releasable state

CFr-.
02018 • "1' u.
Fast and reliable automated testing

Build a fast and reliable automated validation test suite:

o Unit tests
v Acceptance tests
:J Integration tests
>-
1;;
o
u
~
o
E

Tt.sC Pyramid. M/~ COh"

CF1\ UU1~111c
02018 . "'l'.'
Fast and reliable automated testing

Good practice
J Catch errors as early in the automated testing as possible
o Ensure tests run quickly (in parallel, if necessary)
J Write automated tests before writing the code ("Test-driven
development")
J Automate as many manual tests as possible
J Integrate performance testing into the test suite
J Integrate non-functional requirements testing into the test suite
..) Pull the andon cord when the deployment pipeline breaks

CFr-.
02018 • "1' u.
Fast and reliable automated testing

Catching errors as early in the testing as possible always pay back later in
the process.

8S~

_ -.Defeas
found"
thS ph.lst
- sCostto

,_oJ!
....... d<!fea
" Ihk ph.lst

I $100

COding unlt Funrtlon

Test Test

Applied So/cw(Jre Measuremenr, by Copers Jones, .1996

02018
eFr-. unlSll"
• "'1' HI
 Automated continuous deployment & provisioning

Automated Continuous Deployment and Provisioning covers deployment of application

components as well as provisioning of environment components.

Applications &
services
- - (API) Self Servfce - -

Integration Deployment PaaS (OS. Mkldle'WMt' &

Oevelopmt'nl Runltmc)

• •••• •
Continuous
Automated
Provisioning
•• laaS (lnfr...
sttuctut~)

I> Application component Ul Environment component

(I.e. mkrcservlcej

02018
cr-1'. UU1Sl11c
• "'1' HI
Automated continuous deployment & provisioning

Automated continuous deployment automatically push application components into

production when all tests are passed.

Application deployment implies:

J Installing applications
J Updating applications
J Configuring resources
J Configuring middleware components
1 Starting/stopping components
J Configuring the installed application
J Configuration systems like load
balancers. routers
J Verification of components

CFr-.
02018 • ""l' u.
Automated continuous deployment & provisioning

Automated provisioning is defined as the fully automated deployment and

maintenance of environment components.
J Application environment components are the deployment target containers of the
application, such as a database server or an application runtime server.
J Infrastructure changes are viewed as code (Infrastructure as Code)
) Rather than provisioning and managing environment components manually,
DevOps teams must be able to acquire new environment components on
demand, fully automated.
J New environments are delivered within minutes/hours instead of weeks/months,
hence, great speed
) Control, traceability of system changes, no manual changes, and a single,
central source of truth

CF!'.
02018 • "'1' u.
Automated continuous deployment & provisioning

Good practice
J Standardize the platforms and reduce the number of variations
J Automate the deployment and provisioning processes using the same
deployment mechanism for every environment.
J Enable automated self-service deployments for both build, test and deployment.
J Automate as many of the manual steps as possible, such as:
j Packaging code in ways suitabte for deptoyment
J Creating pre-configured virtuaJmachine images or containers
• Automating the deployment and configuration of middleware
) Copying packages or files onto production servers
) Restarting servers, applications, or services
j Generating configuration files from temptates
j Running automated smoke tests to make sure the system Is working and correcUy
configured
) Running testing procedures
) Scripting and automating database migrations
02018
eFr-. • "1' u.
Automated continuous deployment & provisioning

Decouple deployments from releases

J Deployment is the installation of a specified version of software to a given
environment
J Release is when we make a feature (or set of features) available to all our
customers or a segment of customers

eFN
02018 • "'l' ••
Automated continuous deployment & provisioning

Release patterns
environment-based release patterns: This is where we have two or more environments that we
deploy into. but only one environment is receiving live customer traffic
J The simplest pattern is called btue-green deployment. In this pattern. we have two production
environments: blue and green. At any time. only one of these Is serving customer traffic
J The canary release pattern automates the release process of promoting to successively
larger and more crilical environments as we confirm that the code is operating as designed.
When something appears to be going wrong. we roll back: otherwise. we deploy to the next
environment.
) The cluster immune system expands upon the canary release pattern by linking our
production monitoring system with our release process and by automating the roll back of
code when the user-facing performance of the production system deviates outside of a
predefined expected range
Application-based release patterns: This is where we modify our application so that we can
selectivety release and expose specific application functionality by small configuration changes
) tntroduce feature toggles. which provide us with the mechanism to selectively enable and
disable features without requiring a production code deployment. e.g. to enable dark
launches.

02018
eFr-. • "1' u.
Measurement and feedback

Telemetry can be defined as an automated communications process by which

measurements and other data are collected at remote points and are subsequently
transmitted to receiving equipment for monitoring.

Pervasive production telemetry in both the code and production environment ensure
that problems are detected and corrected quickly:
J In applications
J In the environment
) In the deployment pipeline

Key DevOps indicators

J Lead time (request to fulfilment)
J Process time (begin work to fulfilment)
J Percent complete and accurate (%C/A)

eFN
02018 • "1'_.
Measurement and feedback

Monitoring framework
J Collect data at the business Event Routt, ...
logic, application, and
environments layer
J In each of these layers, create
telemetry in the form of events,
logs, and metrics.
J Establish an event router
... Appl;"t.on
responsible for storing events
and metrics
) This capability enables
visualization, trending, alerting
... OperatIng System

Monitoring Ft.mtwork
anomaly detection, and so forth
The Arr oj Monitoring, by James Tvrllbull, 20J4

02018
erN • "'l' UI
Measurement and feedback

Good practice
Create telemetry to enable seeing and solving problems
.) Create a centralized telemetry infrastructure
.) Create application logging telemetry that helps production. Dillerent logging levels, some of
which may also trigger alerts, such as Debug, Info, Waming, Error, Fatal
.) Create the infrastructure and libraries necessary to make It as easy as possible for anyone in
Development or Operations to create telemetry for any functionality they build
J Create self-service access to telemetry and information radiators
Analyze telemetry to belter anticipate problems and achieve goals
J Use means, standard deviation and anomaly detection techniques to detect potential
problems.
Enable feedback so Development and Operations can safely deploy code
.) Use telemetry to make deployments safer
) Have developers follow work downstream, One of the most powerful techniques in
Interaction and user experience design (UX) is contextual inquiry. This is when the product
team watches a customer use the application in their natural environment, often working at
their desk.
Have developers initially selt-manaqe their production service
J

02018
eFr-. • ""l' u.
Measurement and feedback

Human feedback
The key tool that leaders should use and stimulate Is human feedback. Giving and
receiving feedback is the basis for all improvement and development of teamwork. It
is vital for both leaders and team members to leam and practice giving and receiving
feedback in a respectful manner.

Give feedback Receive feedback

Describe specific observations Listen without interruption

Explain what it does to you Avoid discussions or excuses

Wait and listen to clarifying questions Check if there is clear understanding

Give concrete suggestions OR Recognize other person's position

recognition / incentive Thank him/her
Determine whether the feedback is
applied

CF1\
02018 .. "'1' UI
Continuous delivery tools

-..
Om
..
-·
Lb
.

..
~--
~-.-
... .. ,-...
.. .. .. ..,_ ·.. ·..
~

(-, "-
.--
PERIODIC TASLE OF DEVOPS TOOLS evn

._ •• .--
.- .e-.
.-.. . .--
.....
.-
.-
.-
.... ..
.
I_
.~ - Ch

-
Ot
..
.. "".
,
·v.
·An
.. -
SI
. .
.
,
Ok Az

...-.. . .. ..-. --
BI Rk
-
Ge
Tf
- '
Aws

.,'

~
.. - .. .. .. ·-
Mv G,
-
.. . -
At Fn
. -- Se
.. . ..
_... . --
G. Oh
..- ...- -
Jn Ba T,
..
Gd Sf Cn Be
'

Mo Rs
_._-
· .. .. .. - .. .. ,~

-· . '. . . ... ·. ·.. -'. . - - .. .. .--- . . -- . ·.... .. ..-..

Dt Gt Gp B,
"
Cu Cl Ou Npm Cs
.,
Vs C,
~, Cp Ju Rd-. cr, Ds Op
--'-

-· "'sb
Dp
..
Sb
.. Mk
. .. ..
Ck
,_
- ·-- ..- .. ., ..-- .- ..-. - .. - .. --
Jt
· ..
Jm Tn Ay Tc Sh
- . ,,-
Cc
.. .. -
Ry
,
Cy Oc No Kb H,
..
.
- - - - ',_ - - - - -
"

- --
Id Rk Pk
,_ Me Km Jm N, CO Eb ~
Ca
'-'~
So Xid Ud Nm Os
-_--

XebiaLabs
Mii 5 •

Source:hrrps·//xebia/obs.comloeriodic-robte-o(.devops·roots/

CFN UU1~111c
02018 .. "'l'.'
Continuous Delivery Maturity Model

NJ c:nvfnmmllnu
~"", ••'.:i.'I~~'
Pn;.wiIQllina
lI~tCd.
II .r.""y
O~l'Jllof\SlIr'Id
('f8U .,tyeob'~a
tOnM~C!rlskSMld
It' m::,m()ufl
'''''''''':1::
.... """"''d'
fideiSf' to(C!'~.ast
fHdbtdtklOpcl
d31~
~Orm.),n((' ~nd
Vi"~II~uog~edIf rt'duce cyde time. lely. deployment pfOCe$S.
~P9lti •

Oiltllbdse
u~
Envlf01lme1'ltand Dndto Ibat ~ted
~ttIM:M,.,ttd ~~~~~
rr:=n:ii,j,NM
0ev1l101M!1l(hl!(k In

_~~"m. rm:t:~
I!9I~fI'Iel'lts aWlicatlon he-31th withewtv
lNlIar, •RtIN$(' monilor~:lnd fundfonll d~~;L to
prOK! tv
an rollbacl(
~tC!Sled.
periorm;J.ncc lk.7':I~:lls~
=..
lor . mC!~$urC!d. monW;lr~""d
091mlltd.
Au.tDmlltt'd build IMld
10'" CVt~ CYII~ tin14:

~i
Ol!~d(!nc
butt~=ror
:;w=:~~Cha'iUfftJI~m~t
diCW)'
tw~rc,
me prOttiS to
Automillt'dunit.a;r
~.''''
Rtt: OIYoind ..
Pt· app~t:rd aott"l)1oln<clem,'
wi Ilttetwtitll!l'I
t~,e,s. T~tiltfI»orl
o'develo~.t
..
,. ,
Dall1b~clg,n~s

"'tgr~t;,i:~,f'"
"~'" ma:!£
nd
. mlon
con tot uSlt!!
P01"tdN~mj~
ckl",e
milnill JtH~Or ploy 10 Il'ItfY (~. nsmH. process. process. mll"1I8('mlll'll
SCripts 111M!1 • erMronment. con proce$$.
Automilte<! VeNloncon'~ In
Re~liU .utom.:n~ dtPloymcnt lOSOrne: Pilirifuland
bUI::,,,,ndlCS;tins· &~I~,:rn~wreliable.re'e.~.
infr~uf!nt. oot Aj,lIOfNited ttsU d~l
..~t~83J:
~" r=i~~t:m,~'e
Arrt lidcan be fII,~.
crco<llttd'fOro SoOUrce etlYJr:~w.eWIS
control usfna
autolNtod proc~). (On~r~tiOtI
L1rnlte-d
tfKll'able
rro'1~~rJ~~n"
wrlUefi#1IW1of
story dQYClOptl"lC!lu. ~=~ntts
IIppic.uiOl'I,
software: source
~~ldco~r;llion.
Uwi~~~~OY
tlIletilaited I
Yenlon~. rnli"'~!ons,
MDn~proc:cu 'Of
1:~lpr=: fOf ~IIP'oyjn, soft~r~. Ve,siol\ cOlurot
u Idliia twUCl. nY'r~a~';ge<i c Inrt~etlt lIind M.nu~testll'lllllftH Oatil!~18T'Jrn'S eithoerl"l()t
uJtod,Of
~~rnq1'lto' unreJllIbietek'UM. un""" on ",nd
cHNefOl)p)el"lL performed
iltUr~ilnd II'IOInu,jity. ct"~(k·iM I\lI!i)ttI
lnfrtcluenlJV.
rel)Oft$, E:~?1
mallv' Iv.
Ol'!iffly. R(IioO/t SO!IW'b,t~
COIW;nl100 1J\ro4Io1l8uiJ4.
1m ond ~nf AIIlOtnOliI)n.&y JtJHl.Im&k ondDt1rid 1001ey.2OJ~
eFN UU1~111c
02018 "'l'.'
Principles, Concepts, Practices and People

Flow 1. Product 1. Continuous 1. Customer centric

2. Feedback 2. Value stream integration 2. End-to-end
3. Continual 3. Loosely-coupled 2. Fastand reliable responsibility
learning and architecture automated 3. Collaboration
improvement testing
4. Autonomous 4. Learningand
teams 3. Continuous and improvement
automated
deployment / 5. Experimentation
and risk taking
provisioning
4. Measurement
and feedback

02018
eFr-. '1' HI
Customer-centric

It is imperative to have
short feedback loops
with real customers
and end-users.
Therefore. all aotivlties
involved in building IT
products and services
should revolve around
customers.

Eight copabllitles for cusromer-(enrd, orgonilOrloM,

by Mfchoi!1 Hlnsho\v

CFN UU1~111c
02018 . "'l'.'
Customer-centric

Good practice
J Begin with end in mind, in retrospectives: ask peopte why?
J Start off with defining customer objectives (the Voice of Customer). A common
understanding of the Voice of Customer (VoC) is important because in later
stage you will determine with the team, which activities are really adding to this
VoC and which steps are not.
J Encourage customers to attend demos, implement user feedback into a
storyboard, allow customers to write about product and respond, organize people
around the product, encourage the team to write blogs about product (you build
it; you run it).

CFr-.
02018 • "1' u.
End-to-end responsibility

In a DevOps organization, teams are vertically organized so that they can be fully accountable for
the producls and services they deliver. End-lo-end responsibility means thai the learn holds ilself
accountable for the quality and quantity of services it provides 10its customers.
,
... ----.. , ,
...----, 1
,----- .. , r .. ----, I
I Product I I Product I I Product I I Product I
I team I I team I team I I team I

Design

Develop Product o\vner

Integrate Oevelope-r
~
...
Operattons
I I engineer
I I

I I
Test
I I
i,
QA/Tes-tcr
!
In(oSee:
Deploy
f I I I I Release managcr
Deliver End-to-end responslbta
, , , , ,-----;1'
02018
eFr-. . ..,...
(.AU1~111c
End-to-end responsibility

Good practice
Caring about the end-to-end responsibility might be the most crucial ingredient for
DevOps. When people care and have the required skills, knowledge, and resources,
they can and will collaborate to live up to their responsibility. If they care, they will
learn, adapt, improve, and provide great services and value.
'J All team members are responsibte for the complete product, which includes the
full delivery cycle as well as operatinglproviding customer support throughout the
lifecycle of the product.
) Quality is built in from the initiation of the teams up to discharge. It is at the heart
of every activity. It is never compromised. We value full transparency.
) Encourage the team to utilize their skills as they know the best; avoid cutting
corners; practice automation (test, deploy, and provision), continuous
improvement, and transparency (monitors).
J Do not explain 'how' to do, ask 'what' is required, address derailment openly, let
people figure out how to do things, reward responsibility, reward failure, bring
transparency in what everyone is doing.

02018
eFr-. • ""l' u.
Collaboration

The meaning of collaboration is working together to achieve a goal. It is the central

theme for DevOps teams.
Collaboration means shared organizational goals, empathy and leaming between
different people

CFN
02018 • "1'_.
Collaboration

Good practice
Visual Management is one of the strongest tools to stimulate collaboration and
ensures that pitfalls are uncovered. The tool helps ensure the work done by a team is
constantly visible on boards including:
J Identify work and impediments.
) Communicate important
information.
J Show how to perform a task. Day board Week board

) Show planning and priorities.

\
Problems
} Problems
In its most basic form, Visual
Management includes three
boards, as shown in the figure.
<. -~tf../
-fT\_j~
Improvement
board
CFr-.
02018 • "1' u.
Learning and improvement

When we work within a complex system, it is impossible for us to predict all the
outcomes for the actions we take
To enable us to safely work within complex systems, our organizations must become
ever better at self-diagnostics and self-improvement and must be skilled at detecting
problems, solving them, and multiplying the effects by making the solutions available
throughout the organization.

r ~
Dev ~ Ops
..... :/

CF1I.
02018 • "1' u.
Learning and improvement

Good practice
Continuous improvement is about problem-solving:
o Seeing and prioritizing problems
, Uncover problems
, Accept problems as a part of daily life.
_, Initiate an action to identify the problems that need immediate solutions
_, Solving problems
, Invest time and other resources
) Understand the root causes of problems
.> Resolve the problems completely
) Sharing lessons leamed
) Schedule blameless post-mortem meetings after accidents occur. The better Question
to focus on is. Why did it make sense to me when I took that action?"
) Share the lessons learned with others in the IT organization, so they can benefit from it
_, Decrease Incident tolerances to find ever-weaker failure signals, by amplifying
weak failure signals
) Because we care about quality, we even inject faults into our production
environment to learn how our system fails in a planned manner
CF!'.
02018 • ""l' u.
Learning and improvement

Good practice
J Create a single, shared source code repository for our entire organization
J Use chat rooms and chat bots to automate and capture organizational
knowledge. Put automation tools into the middle of the conversation in chat
rooms, helping create transparency and documentation.
~ Reserve time to create organizational learning and Improvement. Dedicated
rituals for improvement work has also been called spring or fall cleanings and
ticket queue inversion weeks. Other terms have also been used, such as hack
days, hackathons, and 20% innovation time.
C) Enable everyone to teach and learn
) Create internal consulting and coaches to spread practices
J Furthermore, everyone is constantly learning, fostering a hypothesis-driven
culture where the scientific method is used to ensure nothing is taken for
granted.

02018
eFr-. • "1' u.
Learning and improvement

Good practice
J Automate standardized processes in software for re-use
J Design for operations through codified non-functional requirements
J Spread knowledge by using automated tests as documentation (test-driven
development) and communities of practice
J Build reusable operations user stories into development
J Convert local discoveries into global improvements

CFr-.
02018 • ""l' u.
Experimentation and risk taking

Experimentation is to test a hypothesis. In other words, trying something new based

on a need is known as experimentation. DevOps teams must have the courage to
experiment. with the potential of failure. They know how to fail fast, take the next step,
or go back a couple of steps under time pressure to ensure there is quality at the
source .
.J Ensure customer buy-in
J Define and deliver a Minimum Viable Product (MVP)
J Focus on Ihe goal "customer value is delivered first time, right in flow"
J Take small steps and do not carry out large experiments

THERE IS NO INNOVATION WITHOUT

EXPERIMENTATION

CFN
02018 • ""l'JI.
Experimentation and risk taking

Good practice
J Create an environment in which people perform at best, where they feel inspired,
where they want to be, feel welcomed and are encouraged to think out of the
box.
J Provide time, use instant sandbox environment, remove hurdles, applaud ideas,
fail safely. Award failure!
J Institute game days to rehearse failures. The goal for a game day is to help
teams simulate and rehearse accidents to give them the ability to practice.
J Inject production failures to enable resilience and learning. For example Netflix
has invented a surprising and audacious service called Chaos Monkey, which
simulates server failures by constantly and randomly killing production servers
J Hire people with matching ambitions, move away from function-title model,
support experimentation, support automating manual tasks, do not focus only on
utilization.
0..) Provide good coffee, create nice and open environments, invest in additional
facilities like games, conduct contests or arrange drinks at the end of the week,
allow teams to modifyltailor the office to their needs.
02018
eFr-. • ""l' u.
Controls

1. Change
management
2. Information
security
3. Separation of
duties

err-.
02018 • "1' U.
Change management

Traditional change controls can lead to unintended outcomes, such as contributing to long lead
times, and reducing the strength and immediacy of feedback from the deployment process.
Fea~essly cut bureaucratic processes.
Enable coordination and scheduling of changes. Even in a loosely-coupled architecture, when
many teams are dOing hundreds of independent deployments per day, there may be a risk of
changes interfering with each other. To mitigate these risks. we may use chat rooms to announce
changes and proactively find collisions that may exist.
Effective change management policies will recognize that there are different risks associated with
different types of changes and that those changes are all handled differently, These processes are
defined in ITIL, which breaks changes down into three categories:
J Standard changes
j Normal changes
) Urgent changes
Re-categorize the majority of the lower risk changes as standard changes
Integrate security and compliance into change approvat processes
Integrate deployment activities with the change approval processes

CFr-.
02018 • "'1' u.
Change management

Enable pair programming to improve all our changes. In one common pattern of pairing. one
engineer fills the role of the driver. the person who actually writes the code. while the other
engineer acts as the navigator, observer, or pointer, the person who reviews the work as it is being
performed. Another pair programming pattern reinforces test-driven development (TOO) by having
one engineer write the automated test and the other engineer imptement the code.

Enabte peer review of changes. Instead of requiring approval from an external body prior to
deployment. require engineers to get peer reviews of their changes.
) Everyone must have someone to review their changes (e.g .• to the code. environment. etc.)
before committing to trunk.
J Everyone should monitor the commit stream of their fellow team members so that potentiat
conflicts can be identified and reviewed.
J Define which changes qualify as high risk and may require review from a designated subject
matter expert (e.g .. database changes. security-sensitive modules such as authentication.
etc.).
, If someone submits a change that you can't understand Its impact after reading through It a
couple of times - it should be split up into multiple. smaller changes that can be understood
at a glance.

02018
eFr-. • "'1' u.
Information security

Instead of inspecting security Into our product at the end of the process, we will create
and integrate security controls into the daily work of Development and Operations, so
that security is part of everyone's job, every day
J Making security a part of everyone's job
, Integrate security into development iteration demonstrations
> Integrate security into defect tracking and post-mortems
o Integrating preventative controls into our shared source code reposttory
) Integrating security with the deployment pipeline
'J Integrating security with the telemetry to better enable detection and recovery
> Creating security telemetry in our appl1cations
> Creating security telemetry in our environment
) Ensure security of the application, the environment and the deployment pipeline
~ Reduce reliance on separation of duty

CFr-.
02018 • ""l' u.
Separation of duties

Reduce reliance on separation of duty

J For decades, separation of duty has been used as one of the primary controls to
reduce the risk of fraud or mistakes in the software development process.
J Separation of duty often slows down and reduces the feedback engineers
receive on their work. This prevents engineers from taking full responsibility for
the quality of their work and reduces a finm's ability to create organizalional
learning. Consequently, wherever possible, we should avoid using separation of
duties as a control.
) Instead, we should choose controls such as pair programming, continuous
inspection of code check-ins, and code review. These controls can give the
necessary reassurance about the quality of work.

CFr-.
02018 • "1' u.
Training and getting started

1. Training 1. Selecta product

2. Further reading 2. Understandand
make the value
stream visible
3. Design
organisation
4. Design
architecture
5. Integrate
operations into
the daily work of
development

eFN C :UI1!\lIlc
02018 "1'_ •
OevOps training

> OevOps Institute (http://devopsinstitute.com/)

>
>
)
DevOps Foundation
Certified Agile Service Manager (CASM)
Certified Agile Process Owner (CAPO)
)
, DevOps Test Engineering (DTE)
) Continuous Delivery Architecture (CDA)
> DevOps Leader (DOL)
J OASA (DevOps Agile Skills Association)
(https:/Iwww.devopsagileskills.orglthe-competence-modell)
D A S 1-\
) DevOps Fundamentals
, DevOps Practitioner
> DevOps Specialization - Enable and scale
, DevOps Specialization - Specify and verify
J DevOps Specialization - Create and Deliver
:) EXIN
(https:llwww.exin.com/en/certifications/exin-devops-master-exam)
, DevOps Master
02018
eFr-. U)l151111
'1' .~
OevOps training - OevOps Institute

START HERE SHARPEN YOUR sxurs BE A GAME CHANGER

DEVOPS FOUNDATION· DEVOPS PRACTITIONER· DEVOPS PROFESSIONAL-

http://devopslnstltute.com/

02018
eFr-. • "'1' HI
OevOps training - OevOps Institute

OASA(OeYOpsAgile Skills Association)

h11ps:llwww.devopsaeilcskllls.org! CFN C :ul1!\lIlc
02018 • "1'_.
OevOps training - EXIN

OevOps Master
J The candidates are expected to
have basic knowledge of OevOps
principles and Lean and Agile
concepts, This knowledge can be
acquired:
J

J
through e-learnlnq:
through an additional training day
•
.,.
'Introduction to DevOps': Or
J by reading The Phoenix Project.
') Training is a mandatory part of the
certification
J 2 day theoretical classroom training
+ 3 day practical training (without
preparation)
J 2 days theoretical classroom
training + 1 day practical training
(with preparation)
htlDS:/lwww.exin.com/en/c.ertificatlons/e>J.ln.devops.masler.exam
02018
eFr-. • "'1' HI
Further reading

J The DevOps Handbook, by Gene Kim, Jez Humble, Patrick Debois and John
Wlilis, 2016
J The Phoenix Project, A Novel About IT, DevOps, and Helping Your Business
Win. By Gene Kim. Kevin Behr, and George Spafford, 2013
J The Lean Startup. How Today's Entrepreneurs Use Continuous Innovation to
Create Radically Successful Businesses, By Eric Ries, 2011
J Continuous Delivery, Reliable Software Releases through Build, Test, and
Deployment Automation, By Jez Humble and David Farley, 2010
Links:

~,
J http://itrevolution.com/
J https:/lwww.devopsdays.org/
J

J
J
hltps:lldevops-research.coml
https:/lconllnuousdellverv.com/
https:lldevops.com/ -'~.
-_
Th.
~nlx

eFr-. (.AU15111c
02018 • '1"~
Training and getting started

1. Training 1. Selecta product

2. Further reading 2. Understandand
make the value
stream visible
3. Design
organisation
4. Design
architecture
5. Integrate
operations into
the daily work of
development

eFN C :UI1!\lIlc
02018 "1'_ •
Where do you start?

5. Integrate

.3. Design
.4.

arganisation
Design
architecture
operations
into the daily
work of
development

• 2; IJnderstand
and make the
value stream
.1.Selecta
product
visible

02018
eFr-. • "1' UI
Where do you start?

1. Select a product
) The age of the application is not a significant predictor of performance;
instead, what predicts performance is whether the application is
architected (or could be re-architected) for testability and deployability.
) Start with the most sympathetic and innovative groups
) Demonstrate early wins and broadcast successes

eFN
02018 • ""l'JI.
Where do you start?

2. Understand the work in the value stream, make it visible, and

expand it across the organisation
J Identify all the members of the value stream who are responsible for
working together to create value for the customers being served.
J Create a value stream map to see the work
) Create a dedicated transformation team with the ability to operate
outside of the rest of the organization that is responsible for daily
operations
J Agree on a shared goal
) Keep improvement planning horizons short
J Reserve 20% of cycles for non-functional requirements and reducing
technical debt
o Increase the visibility of work
J Use tools to reinforce desired behaviour
CFr-.
02018 • "1' u.
Where do you start?

3. Design organisation
o The teams should only be as large as can be fed with two pizzas
o Design the teams to be cross-functional and independent
J Enable every team member to be a generalist
o Fund not projects, but services and products

CFN
02018 • "1' ••
Where do you start?

4. Design architecture
) Ideally, software architecture should enable small teams to be
independently productive, sufficiently decoupled from each other so that
work can be done without excessive or unnecessary communication
and coordination.
) Create loosely-coupled architectures to enable developer productivity
and safety

eFN
02018 • ""l'JI.
Where do you start?

5. Integrate Operations into the daily work of Development

) Create shared self-service capabilities to enable developers in the
service teams to be productive. Create a platform that provides a
shared version control repository with pre-blessed security libraries, a
deployment pipeline that automatically runs code quality and security
scanning tools, which deploys applications into known, good
environments that already have production monitoring tools installed on
them
:) Embed Ops engineers into the service teams or assign designated Ops
liaisons to the service teams when embedding Ops is not possible.
) Invite Ops to Dev stand-ups
) Invite Ops to Dev retrospectives
.) Make relevant Ops work visible on shared Kanban boards

eFN
02018 • ""l'JI.
 TIltCAtM$ ocronym was otlgfnofly dtflfelo(Xtd by John WIllis and Oomon Edwords
In 2010. and toterfurther refined by ter Humble as a means to describe DevOps.

Conclusion

J Culture: It involves current values, beliefs, and attitudes that signify the
corporate environment surrounding and supporting development,
operations, and QA.
J Automation: It is the belief that anything can be effectively automated
to improve processes and reduce errors.
J Lean: It means reducing waste, such as team size, number of tools,
and meeting frequency, to the minimum in order to maximize customer
value.
J Measure: It refers to measuring everything and ensuring there
are mechanisms built to provide visibility into all systems and events,
which should be accessible through a unified interface, like a
dashboard.
o Sharing: It is the necessity of maintaining communication between
development and operations for an effective integration between the
two.
eFN
02018 • ""l'JI.
Questions and comments

CFf\ c :"",,,11
02018 • "1' u.
Christian F. Nissen
CI'l\
cfn@dnconsulLdk
.4S 4019 414S

CfN Consult ApS

UndeAlie 1
OK·2600GloStrup
tv.: 39 364786

02018
eFN c •• ",,,iI
"1'_ •