Faster Smarter DevOps EMag 1512945600237

FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT
Faster,
Smarter
DevOps eMag Issue 56 - Dec 2017
ARTICLE Q&A BOOK REVIEW & INTERVIEW
How to Deal with Merging Designing

COTS Products in Agile and Delivery by
a DevOps World DevOps Jeff Sussna
IN THIS ISSUE
Faster, Smarter DevOps
6 Moving your release cadence from months to weeks is not just about learning Agile prac-
tices and getting some automation tools. It involves people, tooling and a transition plan.
The Things I Learnt About DevOps

12 When My Caar Was Engulfed by Flames
Framed in the story of the author’s car catching fire, this article describes five ways of
thinking to help understand DevOps culture, and behaviours necessary to create an
effective DevOps team.
How to Deal with COTS Products in a DevOps World

16 The most popular agile framework, Scrum, predates the growth of DevOps, and some re-
thinking is required to make the system work in a DevOps environment.
Merging Agile and DevOps

24 Mirco Hering explains why we shouldn’t leave COTS products (and the people working
on them) left behind in a DevOps world. With creative solutions we can apply good prac-
tices from custom software.
Designing Delivery Book Review and Interview

32 Book review and interview with Jeff Sussna, author of “Designing Delivery”, on cyber-
netics, service exchange, customer-centric brands and a new definition of quality in a
service-oriented world.
FOLLOW US CONTACT US
GENERAL FEEDBACK feedback@infoq.com
ADVERTISING sales@infoq.com
EDITORIAL editors@infoq.com
facebook.com @InfoQ google.com linkedin.com

/InfoQ /+InfoQ company/infoq
A LETTER FROM THE EDITOR
Manuel Pais
This DevOps eMag has a broader setting than pre- James Betteley and Matthew Skelton, from Skelton
vious editions. You might rightfully ask, “What does Thatcher Consulting, remind us that Scrum helped
faster, smarter DevOps mean?” Put simply, it means accelerate delivery and, in fact, contributed to the
any and all approaches to DevOps adoption that un- rise of the DevOps movement, as traditional oper-
cover important mechanisms or thought processes ations teams became overloaded. Yet, few organi-
that might otherwise get submerged by the more zations revisited Scrum in light of this new normal,
straightforward (but equally important) automa- where development teams take on responsibility for
tion and tooling aspects. From blending agile and running and monitoring their applications in pro-
DevOps to escaping cognitive biases, planning for duction. They explore in their article moving from
transition, and bringing COTS into DevOps, this eMag product to service backlogs, operability stories, plan-
is a potpourri of insightful practitioner advice. ning for unplanned work during the sprint, and more
techniques for blending agile and DevOps.
Derek Weeks, vice president at Sonatype, wrote the
article that inspired this eMag, in which he talks We shouldn’t exempt business-critical COTS plat-
about the dangers of focusing only on improving forms from DevOps adoption, recommends Mirco
speed of delivery by means of new tooling. Weeks Hering. His article is chock-full of practical advice
highlights the importance of also defining and evolv- on applying version control, configuration manage-
ing a transition plan for an effective DevOps strategy ment, and automated build/deployment even when
that brings everyone on board (bottom up and top lacking direct control over the COTS platform. Hering
down). Shared goals and terminology across teams, illustrates those practices with a real-world example
adequate pipeline metrics, and quality strategy are of a Siebel CRM system he worked on.
some of the key aspects to plan for, while being ex-
plicit in our automation purposes so we can better Finally, Jeff Sussna’s book, Designing Delivery, looks at
sustain and evolve our tool chain. the requirements for successful, holistic service-de-
livery organizations. Aligning agile, DevOps, design
John Clapham, independent coach, trainer, and con- thinking, brand engagement, and a customer-centric
sultant, shares his lessons learned from having his car focus over multi-channel, 24/7 availability systems
engulfed in flames (!) and how that relates to DevOps. will become mandatory to survive. Read the book re-
Our working environments are full to the brim with view and ensuing Q&A to grasp Sussna’s new way of
cognitive biases and knee-jerk reactions — just like thinking about delivery in the service economy.
Clapham was, as he was positive the smoke couldn’t
possibly be coming out of his recently serviced car. These five articles provide plenty of food for thought
In many ways, DevOps requires us to identify and and should trigger valuable discussions in your orga-
explicitly act upon our instinctive “business as usu- nization on the effectiveness of your DevOps adop-
al” behaviors in order to create a high-performing, tion strategy!
blameless culture.
CONTRIBUTORS
Manuel Pais
is a DevOps and Delivery Consultant, focused on teams and
flow. Manuel helps organizations adopt test automation
and continuous delivery, as well as understand DevOps
from both technical and human perspectives. Co-curator of
DevOpsTopologies.com. DevOps lead editor for InfoQ. Co-founder
of DevOps Lisbon meetup. Co-author of the upcoming book
“Team Guide to Software Releasability”. Tweets @manupaisable
Derek Weeks John Clapham

currently serves as vice president and DevOps is an independent coach, trainer, and consultant.
advocate at Sonatype. In 2015, he led the largest and Offering considerable experience in continuous
most comprehensive analysis of software supply- delivery, DevOps, and agile, he helps teams to build
chain practices to date across 106,000 development great products, creating an environment that is
organizations. As a 20+-year veteran of the software effective, productive, and enjoyable to work in. His
industry, he has advised many leading businesses broad experience in software development ranges
on IT performance improvement practices. Weeks from start-up to enterprise scale, formed in the
shares insights regularly across the social sphere at @ publishing, telecommunications, commerce, defence,
weekstweets, LinkedIn, and online communities. and public sector arenas.
Mirco Hering James Betteley
leads Accenture’s DevOps and agile practice in Asia- comes from a development and operations
Pacific, with focus on agile, DevOps and continuous background, which is pretty handy for someone
delivery to establish lean IT organisations. He has over who now works in the DevOps domain! He’s spent
10 years experience in accelerating software delivery the last few years neck-deep in the world of DevOps
through innovative approaches. In the last few years, transformation, helping a wide range of enterprise
Hering has focused on scaling these approaches to organizations use agile and DevOps principles to
large, complex environments. He is a regular speaker deliver better software faster.
at conferences and shares his insights on his blog.
Matthew Skelton Jeff Sussna

has been building, deploying, and operating is an internationally recognized IT consultant and
commercial software systems since 1998. Co-founder design-thinking practitioner. He is known throughout
of and principal consultant at Skelton Thatcher the DevOps community for introducing DevOps
Consulting, he specialises in helping organisations to the importance of empathy. Jeff has nearly 30
to adopt and sustain good practices for building and years of experience in software development, QA,
operating software systems: continuous delivery, and operations, and has led projects for Fortune 500
DevOps, aspects of ITIL, and software operability. enterprises, major technology companies, software-
Skelton curates the well-known DevOps team service startups, and media conglomerates.
topologies pattern.
Read online on InfoQ
FASTER, SMARTER DEVOPS

by Derek Weeks
Call it DevOps or not, if you are concerned about

releasing more code faster and with a higher quality,
the resulting software delivery chain and process
will look and smell like DevOps. But for existing
development teams, no matter what the velocity
objective is, getting from here to there is not
something that can be done without a plan.
Moving your release cadence from months to weeks is not just about learning agile practices
and getting some automation tools. It involves people, tooling, and a transition plan. I will
discuss some of the benefits and approaches to getting there.
Waterfall to agile, agile to continuous integration, continuous integration to continuous de-

ployment — whatever your processes are, the theme is the same: find a way to get code to
users faster without sacrificing quality. But speed and quality are sometimes in opposition.
Going faster means things can break faster, and when we only think about DevOps as releases,
it’s easy to fall into this trap.
6 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Established development shops cannot just jump from one flow to an-
other. Unless you start out net new, the goal is to introduce new process-
es without delaying releases for three months or more and transition
in lump. This is often done with a pincer approach that addresses bot-
tom-up tactics and top-down oversight and culture at the same time.
However, because adopting DevOps tools is so easy, the trend is to focus

on tactics only and adopt from the bottom up without consideration of
the entire pipeline. This leads to release automation tools dictating your
delivery chain for you, and not the other way around. Here are the key
categories that get neglected when teams hit the accelerator without a
plan in place.
Structured automation
DevOps requires automation. But what is often not considered is auto-
mation that sustains and fits into the entire delivery chain. You need to
consider factors such as governance, artifacts organization and invento-
ry, metrics, and security. If an organization establishes a vetting process
for all new automation and how it fits into the pipeline’s orchestration,
then new automation will support what exists today and what will exist
in the future.
For example, many organizations driving down the DevOps path have
encountered challenges when trying to incorporate practices from
security or governance teams. Historically, these teams have resided
outside of the development and operations echo chambers and their
processes were asynchronously aligned to the work being done. The
challenge for many organizations is to determine the best ways to bring
the people, processes, and technology supporting these initiatives into
the fold without slowing things down. The best organizations are find-
ing new ways to automate policies from security and governance teams
by shifting away from artisanal, asynchronous approaches to synchro-
nous processes earlier in the lifecycle.
Let’s look at an example of application security. A number of technology

vendors in the application-security arena are touting automation as key
Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 7

value point for their solutions in order to better fit them into a DevOps tool
chain. In some instances, automation means that machines are now fully
responsible for monitoring, analyzing, and fixing security vulnerabilities
for those applications at wire speed. In other instances, automation eas-
es human workflows that might represent hours or days of asynchronous
analysis not fit for continuous operations. In both cases, the technologies
may accomplish similar ends, but their approaches could be dramatically
different.
Also, one solution might be built to support asynchronous investigations

by a security professional, while the other might provide synchronous
support to a developer at the design and build stages of the systems de-
velopment lifecycle (SDLC). Establishing a vetting process can help deter-
mine if the automation levels required by a team or process can truly be
delivered before making investments. It is also worth noting that layers
of obscurity frequently exist within words like “automation”, “integration”,
“configuration”, and “continuous”.
Common language
Part of the reason you have so many meetings is you are not all speaking
the same language. Even if all understand the other aspects of the soft-
ware delivery chain, it does not mean that teams (QA, development, IT
ops) speak a unified language. And the time wasted to reconcile the dif-
ferences is just vapor. But the solution is easy. Be deliberate. Have a guide
and agree on terminology in the tools you use up front for new aspects of
the pipeline and application.
One approach I have used to establish a common language is to share a

common story. In a recent meeting with a CIO, he told me his aim was to
transform a diverse group of 50 people to operate under DevOps princi-
ples but that they lacked common understanding for starting the conver-
sations. I recommended that he purchase a copy of The Phoenix Project for
each person in the group and ask them to read it. The novel describes the
efforts, challenges, setbacks, and accomplishments of a diverse group of
people as they transform themselves into a DevOps practice.
For example, organizations could map their journey into DevOps similar to
the way the main characters in the book do. They could discuss how their
own organization might take on mastering the “Three Ways”:
• The First Way — understanding the flow of work from left to right as
it moves from development to IT operations to the customer.
• The Second Way — ensuring a constant flow of feedback from right
to left at all stages of the value stream.
• The Third Way — creating a culture that fosters continual experi-
mentation and learning.

Readers can also gain a common understanding of the four types of work
described in the book: business projects, internal IT projects, changes, and
unplanned or recovery work.
Using such stories can provide a foundation for conversations and common
vocabulary that fosters improved understanding, collaboration, and plan-
ning for the journey ahead.
Shared goals
If the entire team — all the players that are a part of software releases —
does not share an objective, then the competing goals will lead to compet-
ing strategies and delayed releases. This results in a Frankenstein pipeline, for
example:
• If development is not accountable for bugs in QA, they will commit fea-
tures faster, but releases are slowed because of new issues. Conversely, if
there is a dedicated QA team that is rewarded for the quantity of issues
found or closed, unnecessary work may be unduly entering the system.
• If IT operations is not motivated by release frequency, they won’t consider
things like full-stack deployments due to perceived risk.
• If security teams rely on automated testing that takes four hours to com-
plete yet the development teams are pushing out new releases every
hour, some application security checks may never happen.
Pipeline and business metrics

If you do not measure the release process (builds, speed, deploys, bugs, etc.),
there is no way to know what to change to increase the speed and quality of
code. But pipeline metrics are not straightforward — and involve the entire
team. The beauty of modern tooling is that it can collect the data and metrics

for you. But it is up to the team to
decide what is valuable. Metrics
available from tools or measured
processes can provide common
ground for understanding what
work is being done or what re-
sults are being achieved. Share
metrics that track work in prog-
ress, deployment frequency, lead
time for changes, mean time to
recover, and things that matter
most to the customer you serve.
Tracking and sharing these met-
rics can help organizations better
understand the constraints with-
in the systems and practices they
are trying to optimize.
That said, you need to focus on

the right metrics, which that
drive business success. Metrics
can help you see if you are pro-
gressing on the things your busi-
ness and customers care about
most — seeing the big picture.
Deming called it “appreciation
of the system”. For example, here
are some pipeline and business
metrics to consider:
• Are you completing activities

(e.g., releasing builds, ship-
ping new features, enforcing
quality, checking security, • Are you spending more or less • behavior-driven testing and
responding to inquiries) fast to acquire new customers or development
enough to matter? support existing customers?
Constantly improving the ways
• What percentage of time are that you catch bugs means your
Quality Strategy test coverage is always increas-
you spending on innovation
Quality is not an afterthought. ing and the number of bugs is
compared to maintenance/
The best development opera- decreasing. It also means, if you
rework?
tions make quality everyone’s re- have a dedicated QA team, that
• What percentage of time is sponsibility and make QA a strat- this team has open communi-
spent on manual/asynchro- egy and automation practice, not cation with IT and developers
nous versus automated/con- an execution one. Then quality and that it has a seat at the table
tinuous activity across your goes far beyond regression test- when it comes time to talk about
development and operations ing and standard unit tests. It quality.
processes? also includes:
• Is your user base expanding
or contracting? Are your cus- • quality of the pipeline, Plans, hurdles, and wins
tomers investing more in your The best way to get from slow to
• component quality and se-
solutions over time? fast consists of these practices:
curity (e.g., binaries, images,
• Are your customers getting tools),
1. Have a plan. Cliché, huh? I
what they want in the time • test-driven development am not talking about a 10-
they expect? (testing new functionality be- page document that you
fore it’s implemented), and put in a drawer. Build a lean

model that covers the prob- some other CI platform. Team proach is more common than
lem, the path to solution, members should be integrat- you might think across the com-
and clear goals. It should not ing their work frequently, munity and can lead to valuable
be static, but built in an ag- and each integration needs insight that you just can’t glean
ile-type way. Think strategy to provide rapid feedback on from online articles or confer-
as code — perhaps taking a the quality and stability of ence presentations.
walking skeleton approach. the build.)
Until you talk/think through In modern development, it is eas-
the entire flow, it’s nearly im- Your environment is as unique as ier to go fast than to make sure
possible to catch all the vari- the combination of all of its indi- that you have the right agility to
ables. And things are always vidual parts. There is no “one size ensure future successes. It is im-
more elaborate than they fits all”. But ensuring attention portant to recognize that taking
seem. If you have ever built to the categories I mentioned the time to build and execute a
a minimum viable product above can help existing opera- plan that enables the right level
(MVP), you will know what I tions move from where they are of agility will be more productive
mean. to a regular cycle of sprints and, than developing a plan defined
eventually, to continuous deliv- only for speed.
2. Clear the hurdles. Don’t ery or even deployment.
think of governance and
change control only after A final tip that I will offer is to
you set up a new process. engage with your community.
Think about it from day one. There are numerous DevOps
And start your automation Days conferences and over
there. Pipelines are only as 1,750 DevOps meet-ups around
fast as the slowest gate. In the world. Right after you finish
many organizations, that is reading this article, find one in
compliance and governance. your area and plan on attending.
So clear your governance The networking opportunities,
hurdles by bringing on tool- open-spaces agenda times, and
ing that can do vulnerability presentations shared there can
checks on your components, be invaluable.
validate licensing, and main-
tain an audit trail of your But don’t just stop at attending
entire build-and-release pro- the events; use the opportunity
cess. to connect with people in your
local area that have already es-
3. Socialize quick wins. Build tablished DevOps practices and
enthusiasm as you start to see if they might invite you into
get results. Address one as- their organizations for a half or
pect of your delivery chain full day of shadowing. This ap-
and make it better. They
should be relatively low risk
and high value. Once you
have done so, socialize the
benefits. These quick wins
get everyone excited about
what is possible. And give
some direction on what to
do next. Continuous integra-
tion — due to the great tools
out there — is a place that
you can start that is signifi-
cant and team-wide and has
a serious impact. (But don’t
fool yourself by just imple-
menting Jenkins, Bamboo, or
Image source: http://devops.meetup.com

KEY TAKEAWAYS THE THINGS I LEARNT ABOUT

DevOps and agile encourage
ways of thinking that are
sometimes unnatural to us and
DEVOPS WHEN MY CAR WAS
work against our instincts.
The same ways of working are

ENGULFED BY FLAMES
often opposed to established by John Clapham
organisational ways of working.
It takes a conscious effort to This is a true story, based on a talk

overcome our cognitive biases to
make the best of our skills, and to from DevOps Days London 2016.
relate to our colleagues.
It was a gorgeous sunny spring day. My family and I were
Our ability to learn and improve
driving through my hometown of Bristol, ready for another
as people (and teams) is often
hampered by a perception that weekend adventure. We were cruising along when my wife
failure is entirely negative. This said quietly, “I can smell smoke.”
view may arise from us or be
inherited from an organisation’s Now, I’m a good mechanic; I used to restore classic cars. The
culture. car we were driving was modern and recently serviced. I
checked the instruments, everything normal. “It must be out-
There are five takeaways in this
side,” I declared confidently.
article, framed in the true story of
the day the author’s car caught
fire…. Two minutes later tentacles of smoke were curling around
my ankles and my shins were getting remarkably warm.

We can learn something relevant
to DevOps, and many other disci-
We swiftly pull the car over. We
are in one of Bristol’s less than Ancillary systems,
plines, from this experience. salubrious areas. In fact, it has a
reputation for tracksuits, hood-
ies, aggressive dogs, caps, and
with their
Don’t let your expertise
be a blind spot
numerous police vehicles. I step
out of the car and I’m immediate- ownership sitting
People who are regarded as ex- ly confronted by a gang of youths
perts in a field are likely to be
called upon to provide answers
who say, “Get your &^%*&^% car
out of here. You can’t park there.”
in the uncertain
quickly. Humans are very good at
this. Instinctive answers engage I reply, “Look, my car is on fire.
I want to get my family out and
space between
what has been called “System 1”,
a rapid-fire but slightly lazy part
of the brain. It tends to reach for
then we can talk.” The change
in attitude was astonishing. “I’ll
Dev and Ops, are
instrumental to
the easiest answer to hand, useful fetch a hose,” says one. “Do you
in a fight-or-flight situation, less need a hand?” says another.
helpful in a meeting of minds.
System 1 is designed to front
our System 2, which is relatively
What can be learnt from this? As
DevOps practitioners, we often rapid recovery
[from disaster].
behave in ways that seem uncon-
rational but needs time to fig-
ventional, baffling, or threatening
ure things out. The relationship
to others. From outside the car,
between Systems 1 and 2 is like
the lads couldn’t see the fire so
the relationship between a cache
they couldn’t guess our context.
and a server request. Sometimes
What they saw was an ordinary
though, when our mental cache
car roar into their space and a
can’t find an item, it offers the
stressed stranger jump out. I took
quickest thing it can find instead
time to explain the situation and
of calling the server. Many times,
they listened. Once they under-
our quick, instinctive answer,
stood the situation (and my moti-
based on years of experience, will
vations), they made a judgement
be correct. Other times, probably
call to help.
when it’s most embarrassing, it
won’t.
People are more
In problem-solving situations and likely to assist when
thought work, we need to learn they understand your
to anticipate and reflect on this motivations
initial, instinctive answer. A useful This is something I see frequently
technique is to challenge yourself when people are promoting new
to look for a second answer, to try techniques, be it agile, DevOps,
to refute your first. As we leap to or a fresh technical approach.
the nearest conclusion, we often The new concept makes perfect
ignore useful input from others, sense to the promoter — in fact,
particularly if we regard them its merits are so obvious that the
as less knowledgeable than our- promoter doesn’t explain con-
selves. text and motivation to potential
adopters. Unable to understand
The curse of knowledge speaks of or develop empathy, people will
a similar challenge: how to relate often resist or blithely continue
to people new to a field. I believe with their habits.
this “curse of being good at stuff”
is similar and more focused on The value of this kind of empathy
the tendency to jump to conclu- cannot be understated. It applies
sions when we feel pressure. both ways: if we as promoters of
change expect to be understood

We should also and listened to, then we too must
understand the feelings and chal-
chat servers, something they
relied on to understand system
lenges of others. state and to collaborate in the
inspect our So the family are clear of the
event of an emergency.
own reaction to car, the heat has intensified, and

flames are coming out of the bon-
We call the fire brigade, which
thankfully arrives promptly. The
net. It’s time to put the fire out. team of four exchange very few
failure. Do you I go to the boot where the extin-
words yet move swiftly with a
sense of purpose, each member
look upon them

guisher is stored. It doesn’t open. contributing in a different way.
The molten blobs of electrical Hoses are selected, traffic is redi-
insulation under the car provide rected, a water supply is found,
as opportunities a clue as to why. I take a deep
breath, dive into the car, and grab
people are directed to a safe dis-
tance, and the hoses are turned
to learn? Or are
the extinguisher. With the flames on. My car is drenched in gallons
growing, I unwrap the extin- of water. It’s clear that even if I had
guisher and leap into action — by practised, all the fire extinguisher
they explained reading the instructions. I fiddle
with the pin and point the extin-
would have done is buy us time.
“You need to understand the fire,
guisher, but the foam isn’t going
away with self- in the right direction. Just as I get
it to spray in the right direction, it
see?” says one of the fire fighters.
“It’s deep in the bulkhead, very
hard to reach.”
comforting logic? sputters and stops.
So how are fire fighters so good at
My fruitless attempts to use a tool what they do? For one thing, they
with which I’m not familiar at the follow the previous advice: they
time it most matters tell us some- practice for disaster and they are
thing about our approach to un- well rehearsed. Fire fighters also
expected situations: learn quickly. One way they learn
is through investigation of fail-
ures.
Don’t just plan for
disaster but expect it,
practice for it Failures are rich in
Operations groups are pretty learning
good at planning for disaster, and The approach of Western cul-
disaster-recovery plans are rec- tures to failures appears to need
ognised good practice. This calls a massive rethink. The DevOps
to mind the classic boxing quote: and agile movements lead the
“Everyone has a plan until they way, encouraging recognition
get punched in the face.” Typically, that triumphs and failure alike
efforts focus on production sys- are opportunities to learn and
tems. These alone are often not improve. You are likely to be fa-
enough. Ancillary systems, with miliar with root-cause analysis,
their ownership sitting in the un- retrospectives, and post mortems
certain space between develop- (blameless and otherwise). These
ment and operations, are instru- all encourage learning from re-
mental to rapid recovery. There cent events and commitment to
are plenty of stories of systems making improvements.
recovering their compute, but be-
ing unable to recover further due We should also inspect our own
to their configuration-manage- reaction to failures. Do you look
ment or deployment-tool servers upon them as opportunities to
not being available. The duration learn? Or are they explained away
of GitHub’s January 2016 outage with self-comforting logic? I often
was increased by the loss of their think of Danny MacAskill, a Scot-

tish trials bike rider. He practices the tried that didn’t work as well as those
same jump over and over, falling hard that did. Science has long recognised
on many occasions, until he gets it the value of this approach and many
right. Each time, each iteration, brings tangential findings have led to useful
him closer to his goal. applications. Blogs, kanban boards,
and open meetings are all examples
The approach organisations take to of sound sharing practice. All these
failure is so significant that Westrum’s invite serendipity and encourage oth-
typology of organisational culture ers to build on what you do.
uses the treatment of messengers
(those who often bring news of fail- So that’s the story of the day my car
ure) as an indicator of the type of cul- caught fire and the things I learnt
ture an organisation has built. Where about DevOps from the experience.
reporters of failure are encouraged to It may be instructive to read those
speak up, their concerns investigated, lessons once more and note that they
and corrective action taken, you’ll of- are highly applicable outside of the IT
ten find a high-performing organisa- world. This is one of the elements that
tion. draws me again and again to DevOps.
Through it and the willingness of the
Finally, we are back at home. We try community to share, we may learn
not to think too hard about what hap- skills that serve us well in other as-
pened, or what could have happened pects of life.
in very slightly different circumstanc-
es. We upload a few photos. Friends
sympathise and make summer BBQ
jokes.
Months later, we are contacted by an

owner of the same type of car. It tran-
spires that this model has been suf-
fering spontaneous fires all over Eu-
rope. None of the drivers know each
other, but there are enough to form a
community, enough to support each
other, enough to get the attention of
the manufacturer, enough to engage
a lawyer, enough to prompt a glob-
al manufacturer to issue a recall. All
because a few people shared their
photos, and cared about the cause
enough to connect with each other.
If it matters, share it — you

never know who’ll benefit…
and it could be you
Sharing is another tenet of DevOps:
share knowledge, code, metrics,
styles, approaches, and patterns. Of-
ten though, we share what interests
us, what we’ve been asked to, or what
process demands. I encourage what
I term “deliberate sharing”: the act of
sharing things because they might
be interesting or having your default
sharing setting as public rather than
private. This includes the things you

KEY TAKEAWAYS HOW TO DEAL WITH

Configuration management is
the basis for any good DevOps
adoption as it is crucial to enable
COTS PRODUCTS IN A
speed.
COTS products will continue to

DEVOPS WORLD
be relevant in the DevOps world by Mirco Hering
as they continue to support key
business functions.
Version control for COTS

The primary objective of DevOps
products requires creative is to increase the speed of delivery
solutions to identify relevant
code and store it in common with reliable quality. To achieve this,
source-control tooling. good configuration management is
It is possible to significantly crucial, as the importance of the level
reduce effort by treating COTS
code similar to custom code. of control grows with higher speed
of delivery (while riding a bike, you
Four steps will make your COTS
solution more manageable in the might take your hands off the handle
DevOps world by making it easy
for COTS developers to do the bar once in a while, but a Formula
right thing. One driver is practically glued to the
steering wheel).

Yet commercial off-the-shelf
(COTS) products often don’t pro-
Besides limiting speed, that COTS
product might also soak up a lot Look for
vide any obvious ways to man- of effort to support several code
age them like you manage your
custom software. This is a real
lines at once (production mainte-
nance, fast releases, slow releas-
opportunities to
challenge for large organisations
that deal with a mixed technol-
es), which has become a com-
mon pattern in organisations. The make new practices
ogy landscape. This article will efforts required to branch and
explore ways to apply modern
DevOps practices when dealing
merge code and do the required
quality assurance increase with
easy to adopt. Good
with COTS products. each code line. I have seen code-
merge activities consume up to processes that are
difficult to follow will
20% of the overall delivery effort
COTS products are and add weeks and sometimes
an important part of months to the delivery timeline.
enterprise landscapes
Why should we even deal with
My experience indicates that this
merging effort can be reduced
hardly be followed.
COTS and other systems of re- by up to 80%, saving millions of
cord? dollars. This figure was calculat-
ed by comparing the proportion
Consider the analogy of two of effort for configuration man-
gears. agement before and after the
implementation of the practices
If you can completely ditch your outlined in this article.
legacy applications, then con-
gratulations — you don’t have to
deal with this and can probably Many COTS products
stop reading this article. The rest have not yet shifted to
of you who cannot do that will the DevOps world
eventually realize that while your You might wonder whether COTS
digital and custom applications vendors have understood the
can deliver at amazing speed need to operate in a world fo-
now, you are still somehow con- cussed on DevOps and continu-
strained by your legacy applica- ous delivery. In my experience,
tions. Speeding up the latter will there is a realisation that this is
help you achieve the ultimate important but most of the solu-
velocity of your delivery organi- tions provided by the vendors are
sation. not yet aligned with good prac-
tices (like bespoke SCM solutions
For example, an organisation and the lack of development-tool
uses a COTS product for their cus- APIs). The guidance I offer below
tomer relationship management falls mostly in a grey area where
(CRM) to provide information to vendors don’t encourage you to
both a digital channel (like an use these methods (as they pre-
iPhone app) and to their custom- fer you to use their solution) but
er-service representatives (CSRs), you are not breaking anything or
and the speed of providing new compromising your support ar-
functionality on the iPhone app is rangements. As a community, it
limited by the performance speed is our responsibility to keep push-
of the back-end CRM system. In- ing COTS vendors to adopt tech-
creasing the delivery speed of the nical architectures that better fit
CRM system in this case speeds a DevOps context. In my experi-
up not only the enablement of ence, the feedback has not been
the iPhone app but also puts new strong enough yet and vendors
functionality in front of the CSRs can continue to ignore the real
more quickly.

needs of DevOps-focused organ-
isations.
I would love for vendors to reach

out to our community but so
far I have not seen this happen.
It is up to us in the industry to
demand that they do the right
thing or alternatively to start How to approach no fan of branching and merg-
voting with our feet and slowly COTS code and get ing, more often than not it is a
move away from their misfit solu- its configuration necessary evil that you need to
tions. When I have the choice and management under deal with. In my experience, this
it is economically reasonable, I control process can be extremely costly
avoid introducing new applica- and error-prone with COTS prod-
tions that don’t meet the follow- Step 1: Find the source ucts and improving this alone
ing minimum requirements for code will lead to meaningful benefits.
DevOps: COTS applications can be pret- I have seen organisations that
ty annoying when it comes to tracked which modules were
• Can all source code, configu- finding the actual source code. changed in a release in an Excel
ration, and data be extracted Many of them come with their sheet, and their merge process-
and stored in external ver- own configuration management es required comparing those
sion-control systems? and vendors will try to convince sheets, followed by some manu-
you that those are perfectly fine. al activity to resolve the conflicts.
• Can all required steps to No, not just fine — they are more Not only is this error prone, it is
build, compile, deploy, and appropriate for your application also quite labour intensive. By
configure the application be than industry tools. They might being able to store your code in a
triggered through an API (CLI be right, but here’s the thing: standard version-control system
if programming-language it’s very unlikely that you only you can reduce the error rate to
based)? have that one application and nearly zero and achieve a reduc-
• Can all environment configu- it’s very likely that you want to tion of effort of up to 95%.
ration be exposed in a file that manage configurations across
can be manipulated program- applications. I have yet to find a The table above shows an exam-
matically? proprietary configuration-man- ple of the effort required for a
agement solution that can easily single merge activity
integrate with other tools.
So what can you do if you don’t
Imagine a baseline of code. You want to use the proprietary
want to be able to recall/retrieve source-control tooling? First of
the configuration across all your all, identify all the components
applications, including source that your application requires.
code, reference data, deploy- The core package and its patch-
ment parameters, and automa- es are better managed in an as-
tion scripts. Unfortunately, this set-management tool so I am not
has so far not been possible for going to discuss those. What you
me with COTS configuration want in your configuration-man-
products. They also usually don’t agement tool are the moving
track all the required changes parts that you have changed. For
very well, but mainly focus on a example, in a Siebel implemen-
subset of components. tation I was dealing with, the
overall solution had over 10 000
Last but not least, they poorly configuration files (once we ex-
deal with parallel development posed them) but our application
and the need for branching and only touched a couple hundred
merging. While I am certainly (about 2% of all files).

Storing all the other files will on the fly. For example, for one identify that the file already ex-
just bloat your configuration of our Siebel projects we creat- ists somewhere else in version
management with no real ben- ed a little custom UI in .Net that control. Controlling the location
efit, so try to avoid it. Especially intercepted any change made of files via the mechanism de-
when you later want to run a full through the Siebel Tools IDE, scribed above solves this prob-
extract and transfer, this can be- thus forcing a check-in into our lem also.)
come a hindrance and the signal- version-control system with the
to-noise ratio gets pretty low. If required metadata. This UI used In Figure 1 you can see the cus-
you measure percentage of code the temporary storage of the tom IDE, which supported:
changes between releases, this Siebel Tools IDE to identify the
is only meaningful if you anal- changed files and pushed them • requiring username and pass-
yse the code that your applica- into the version-control system word for the developer to log
tion changed rather than the full in a pre-defined location to avoid in,
code of the base product. any misplacement of files.
• automatically assigning a lo-
cation for the file,
Once you have identified all the (Note: When manually storing
components that require track- COTS config files in version con- • allowing the developer to
ing, you have a few ways to deal trol, they often end up in mul- search for the right work item,
with them: tiple locations because the re- • allowing check-in comments,
pository folder structure does and
not matter to the COTS product.
Option A: Interfere with the When importing files back into • providing feedback on the
IDE COTS products, only the file status of the check-in.
The most effective and least er- name and/or the file content is
ror-prone way to do this is by in- important, not where the file is Option B: Extract on a regular
tegrating the developer IDE with located on the file system. As a basis
the version-control system in the result, developers will often store Where you cannot easily inter-
back end to intercept any chang- a (duplicate) file in a new location fere with the IDE, you might want
es made to the COTS application when they are not able to quickly
Figure 1: A custom IDE to intercept any code changes.

to use a regular extraction utility solution as it requires additional work items. Integrate any addi-
to pull configuration files out of effort by the developers and in- tional tooling into the natural
the COTS application and push it creases the risk of overwriting re- steps of a developer. For example,
into version control. This could be cent changes in the environment use an IDE that can provide basic
done every night or even more when developers don’t adhere to coding checks (e.g., in COBOL, to
frequently. For the same reasons this process and use the COTS IDE start commands from column 8)
as explained in option A above, instead. For this solution to work, and integrate with a ticket system
you should look for a way to we had to automate the deploy- like JIRA.
identify only recent changes and ment process and keep control
not push every file into version over the environments. Among mainframe developers
control every time. Furthermore, used to developing in a text-
while many version-control sys- based system, the ease of getting
tems ignore check-ins of exact Step 2: Make good feedback this way may improve
copies, the performance of this practices easy for adoption. As mentioned, in Sieb-
solution would be very much im- developers el, you can create steps in the
pacted by the number of files. Many times, developers working IDE that automatically commit
with COTS or legacy applications code into your chosen configu-
are just not used to modern de- ration-management system and
Option C: Force outside velopment practices. Enforcing make it easy to identify the work
creation of files these can feel like extra overhead item(s) you are currently working
A few years ago, I worked with and can make the adoption much on.
some smaller COTS products that harder than necessary. Look for
didn’t allow me to follow either of opportunities to make new prac- All these changes will increase
the above processes as there was tices easy to adopt. developer adoption of appro-
no programmatic hook into the priate practices — not because
UI of the IDE and it provided only For example, don’t force main- they are better for the team, but
import functions, not export. In frame developers to move their because they make life easier for
this case, we changed the process files to a different file system to the developers themselves. Good
for developers to basically devel- check code into your preferred processes that are difficult to fol-
op outside of the COTS IDE and configuration-management sys- low will hardly be followed.
built automation that imported tem.
the files into the COTS product Even obvious improvements can
upon check-in to version control. Don’t make developers switch be hard to implement. At one
This is clearly the least favourable context to use JIRA for tracking company, I was trying to convince
Figure 2

developers to use an IDE for COBOL development rather than using a text pad, as the latter would
only identify basic coding issues (such as a command starting in column 7) once code had been
uploaded to the mainframe. The team didn’t come around to the idea of adopting the IDE until I
proved that my code when uploaded failed significantly less than the average developer’s code.
Step 3: Support intelligent merges

Developers who are used to native COTS products are often not familiar with three-way merges.
Even if they are, traditional tooling might not provide the necessary support. I will showcase this
in the case of Siebel code. We will have to dive a little deeper here to explain the idiosyncrasies of
Siebel code and the Siebel Tools IDE.
When you prepare to merge code natively, Siebel tools basically compare two versions of the file
and show you the differences, without identifying an ancestor. You then have to judge what to
do with it. Siebel tools do not know about three-way merges. Figure 2 demonstrates how a three-
way merge can help identify conflicts.
When trying to use a common configuration-management tool to enable three-way merges for
Siebel, I ran into a new problem. Developers did not trust the configuration-management tool. I
was surprised, but a closer look at the Siebel code showed me the problem. Let me explain this
by first showing you a code sample:
As you can see, Siebel stores some metadata in the source code (e.g., user name and timestamp
of change). A configuration tool that is not context aware will show you a conflict if your files dif-
fer only by timestamp but are otherwise the same. If you open the same files in the Siebel Tools
IDE, which does understand that this difference is not relevant, it shows you no conflict between
the files. If you run a report using traditional configuration management tools, you will see a large
number of false positives.
This leaves you with the choice of Siebel Tools, which avoids false positives but does not pro-
vide three-way merges, or a traditional configuration-management tool that provides three-way
merges but shows a lot of false positives. Here is where better merge tools make all the differ-
ence. Tools like Beyond Compare allow you to define a custom grammar that identifies parts of
the code that are not relevant for the merge. Look for such grammar for your entire COTS config-
uration and use the merge tool that is most appropriate.
Below are the results from my project, which show a significant reduction of merges that required
manual intervention. There’s also a breakdown of the different kinds of file merges.

Step 4: Close the loop by enforcing
correct configuration (a.k.a. full
deploys)
COTS products and other legacy systems of-
ten suffer from configuration drift as people
forget to check code into version control. Be-
cause configuration management is not some-
thing COTS developers traditionally deal with,
the chance is higher that someone makes a
change in the environment directly without
adding code to version control. This means
that the application or environments do not
match what is currently stored in configuration
management. If something goes wrong and
you need to restore from configuration man-
agement, you will miss out on those changes
made directly in the environments. We want to
minimise this risk and the associated rework.
The most practical way to deal with configura-

tion drift is to redeploy the full application on
a regular basis (ideally daily, but at least every
week). This will over time enforce better align-
ment and minimise the amount of drift.
Conclusion
COTS and legacy code can behave a lot more
like your normal custom code if you put some
effort into it. This means that you can leverage
common practices for code branching and
merging, reliable environment configuration,
increased resilience to disaster events, and, as
a result, more predictable delivery of function-
ality to production.
Some creativity is required and the bar is a bit

higher to shift the culture in your development
team but once you get there, the productivi-
ty and predictability of development will pay
back significantly.
In one project, we were able to reduce non-val-

ue-added development time by over 40%. In
another project, the configuration-related de-
fects and outages were reduced by over 50%.
And if that is not motivation enough, the COTS
and legacy teams will be able to work much
closer with your other teams once the practic-
es are aligned and the legacy teams don’t feel
like they have been left behind.
You can all move forward on your DevOps

journey together!

COTS and legacy
teams will be able
to work much
closer with your
other teams once
the practices are
aligned and the
legacy teams don’t
feel like they have
been left behind.

KEY TAKEAWAYS MERGING AGILE

The most popular agile framework,
Scrum, predates the growth of DevOps. In
consequence, the practices within Scrum
AND DEVOPS
(and other agile frameworks) overwhelmingly by James Betteley and Matthew Skelton
focus on what you might loosely define as the
development aspects of software delivery and
focus less on the operational aspects. There’s no point building a
A blended DevOps approach requires some super-cool, super-functional
rethinking of teams, backlogs, how user product that looks and feels
stories are written, and so on. For example, a
backlog should include scalability, deployability, awesome for the customer if
monitoring, and so on.
we can’t deploy it, maintain it,
Sprint planning should include some DevOps and support it once it’s gone
aspects so that we discuss not only product
functionality but operability features as well. live.
A conventional ScrumMaster may not fit well
into this blended approach — the role is more In the agile world, great efforts have been put into
that of an agile coach. making sure we deliver what the customer expects,
within reasonable budget and on time. We also go
We need to consider DevOps right from the to great lengths to help our customers determine
moment we hire our team members, from the
the highest-priority features, so that we shift our
planning and building of our products through
to their ultimate retirement. focus towards delivering high business value. We
deliver early and often to get regular and relevant

feedback. We use user stories to DevOps teaches us that operabil-
help us think from an end user’s ity (that is, operational features)
perspective, and we test our code is actually a first-class citizen, and
on every commit to make sure should be treated with as much
we’re not breaking our codebase. regard as any other product fea-
ture. The best way to ensure this
This is great, but where are all happens is to foster a strong cul-
the clever tricks and techniques ture of collaboration between
designed to ensure we deliver development teams and opera-
deployable, scalable, high-per- tions. How we achieve this collab-
forming products that we can oration is another question, and
update in real time, monitor from DevOps models can differ quite
the very second they’re built, and wildly, from the Amazon “You
manage from day to day without build it, you run it” approach,
needing a team of support engi- where both development and
neers? operational activities exist with-
in a single product team, to the
Agile has borrowed (and contin- “DevOps as a platform” approach
ued to evolve) great ideas from found in some Google teams.
the automotive industry, neu-
roscience, ancient philosophy,
the military, and mathematics, The need for DevOps
to name but a few (think lean Agile and DevOps have lived side
manufacturing, cognitive bias, by side for a few years, and there’s
servant leadership, planning, and been plenty of discussion around
relative sizing). It’s now time to the relationship between the two.
borrow some thinking from the
DevOps scene to ensure that ag- Some people see DevOps as
ile remains the most suitable and a subset of agile, others see
successful set of principles and DevOps as “agile done right”, and
practices for delivering products. others see DevOps as a set of au-
tomation practices, loosely con-
Most products spend the major- nected to the agile big picture.
ity of their lives being support- It all depends on our individual
ed and maintained after they’ve definition of DevOps. Regard-
been launched (bug fixes, feature less of how we see DevOps, the
releases, and enhancements, for intention of delivering working
example). The practical way in software that can be managed,
which we manage these (rolling maintained, scaled, supported,
out changes to a live service, test- and updated with ease is some-
ing in live-like environments, and thing the software-delivery world
so on) and how the product can desperately needed.
scale for performance are seen as
operational features, and are of- The way we run and operate
ten nowhere to be found on the software has changed massive-
product backlog. ly since agile frameworks were
invented. Scrum started back in
A 2014 report in ZDNet cites a 1993, DSDM launched in 1994,
survey from consulting firm CEB, and the XP book was launched in
which “found that 57% of the 1999. Back then, we were writing
budget will go towards mainte- MSI installers, burning them to
nance and mandatory compli- disks, and posting them to peo-
ance activities, down from 63% ple!
back in 2011.” A Gartner report
from 2006 put the figure as high Running, maintaining, and oper-
as 80%. ating software was generally not

Figure 1
something most software devel- natural, and allowed many agile and improved quality as a result),
opers were involved with in any teams to get back to what they but the same cannot be said for
way. felt most comfortable with: de- everyone. Evidently, there’s con-
veloping software rather than de- siderable resistance to adopt-
Since then, a major shift to SaaS livering it. ing new practices into particular
and PaaS has occurred and the agile frameworks, even if those
production environment is at our Unfortunately, this fits the Scrum practices are themselves agile to
fingertips. Developers are now framework quite nicely — the the core. And now we’re seeing
actively involved in the operation development team focuses on the same thing with DevOps.
and support of their systems, but designing, developing, and test-
we’re still following frameworks ing their software and the con-
that don’t accommodate this tinuous-delivery team focuses DevOps anti-patterns
change in the way we work. on managing the system that de- The main focus of DevOps is to
ploys it and the underlying infra- bridge the gap between devel-
structure. opment and ops, reducing pain-
Continuous delivery to ful handovers and increasing
the rescue, almost The trouble with this approach, of collaboration, so that things like
Continuous delivery requires course, is that by separating the deployability, scalability, moni-
deployment automation. This build and deployment-automa- toring, and support aren’t simply
was a step in the right direction tion along with the infrastructure treated as afterthoughts.
— even if it did, in some organ- management, we’re essentially
isations, inadvertently create a making it somebody else’s prob- However, we’ve already started to
spin-off profession of continu- lem from the perspective of the see strong anti-patterns emerg-
ous-delivery engineers (thus of- agile team, and operability once ing on the DevOps scene, such as
ten creating another silo). Con- again disappears into the back- the separation between the de-
tinuous-delivery engineers grew ground. velopment team and the DevOps
into continuous-delivery teams team, effectively creating anoth-
and, eventually, platform teams Many teams did embrace con- er silo and doing little to increase
as infrastructure management tinuous delivery the “right way” collaboration. (Figure 1)
became increasingly central. In and that enabled them to adopt
many cases, this shifting out of a “we build it, we run it” approach The problem is that there is very
the continuous-delivery aspect to software delivery (and with little information from a practical
into a separate team seemed that a greater sense of ownership perspective on how to actually

Figure 2
blend your agile development some of the latest and greatest in each and every agile team, and
teams with this new DevOps ap- DevOps best practices? Well, it’s we might be right, but don’t for-
proach. easy — we just shift left! get people said that exact same
thing about testers, and archi-
What practices do we need to That sounds a lot easier than it tects, and database engineers,
adopt? Which practices do we is in practice, but the concept and UX, and so on….
need to stop? How do we get is straightforward enough. We
started? What roles should we underline this by adding opera- If the way we deliver, support,
have in the team? These ques- bility tasks/stories to the back- update, scale, and maintain our
tions remain largely unanswered. log, alongside our user stories. product is important, then we
As a result, teams are bolting on Our backlog suddenly becomes need these skills in our team.
DevOps rather than fully integrat- a full set of epics, stories, and
ing it into their software develop- tasks needed to get our product Is this going to mean that
ment processes. (Figure 2) delivered successfully and then you have to break Jeff Bezos’s
maintained once it’s gone live (as “two-pizza team” rule? Maybe.
In this classic DevOps anti-pat- opposed to simply a set of func- But if our share of pizza is that
tern, we have all the agile cere- tional features from an end user’s important, we could always skill
monies happening, and many perspective). up! (This isn’t actually as daunt-
of the usual DevOps practices ing as it sounds — the more we
as well, but the end result is no On the surface this might sound move towards an X-as-a-service
better than before — operability easy, but there are a couple of world, the less hard-core sysad-
is still an afterthought and prod- considerations: min knowledge we need. Instead,
ucts are optimised for develop- we’ll all need a firm understand-
ment rather than for delivery and • Who’s going to work on these ing of cloud functions and related
operation. This is all because the operability stories/tasks? services.)
key DevOps practices are being
• How can we write an operabil-
bolted on rather than being pres-
ity story if there’s no end user? Backlogs
ent from the start.
• What are these so-called If we have cross-functional
The solution, of course, is to bake DevOps best practices? teams, then we’re going to need
these good DevOps practices in cross-functional backlogs.
• How can a product owner be
from the very beginning, by ab- expected to manage this?
sorbing them in to our daily agile Leave the traditional view of a
processes and practices — and The answer to all of these ques- product backlog in the past — it’s
this is what requires some tweaks tions is: “Things will need to time for a fresh approach that em-
to our agile frameworks. change.” braces the operability aspects of
our services. And we use the term
“services” intentionally, because
Updating agile practices Teams what we tend to build these days
So what can we do to ensure Most agile teams we work with are indeed services, not shrink-
we’re developing software in an don’t include ops, support, or in- wrapped products. Services are
agile manner, while also deliver- frastructure specialists. We might products that need to be de-
ing and maintaining our products argue that there’s insufficient deployed, scaled, maintained, mon-
and services in accordance with mand for such specialisms to be

We need to consider itored, and supported, and our
backlog needs to reflect this.
to spend a good portion of the
product budget (and team time)
on operational aspects. As a
DevOps right from Most Scrum product backlogs
that we see contain something
rule of thumb, I have found that
spending around 30% of product
the moment we hire like 90% traditional features that

can best be described as a col-
budget on operational aspects
produces good results, leading
lection of desirable features from to maintainable, deployable, di-
our team members, an end-user’s perspective. The
remaining 10% tend to be perfor-
agnosable systems that continue
to work for many years.It should
through the planning mance related or something to

do with preparation (setting up
be noted that these operability
and security requirements are
dev environments, prepping da- continually changing, and evolve
and building of tabases, and so on). The weight-
ing towards end-user functional-
with the product/service, so one
cannot simply get them all done
our products right ity/product features is revealing.
This could be a consequence of
at the start of the release and
then move on to the traditional
through to their
the Scrum framework itself or a product features. For example, it
result of end-user bias by product may not be cost-effective to im-
owners (or something else entire- plement an auto-scaling solution
ultimate retirement. ly). for our system until that system is
commercially successful. Or per-
Instead, a modern service back- haps we need to change our en-
log should describe (besides user cryption model to conform to a
functionality): new security compliance. Equally,
we may need to change our very
• the scalability of the product/ deployment model when new
service (up, down, in, out — geographic locations come on-
and when); line. Also, monitoring will usually
need updating when any sizeable
• the deployability (Does this
change to an application’s func-
need to be deployed live with
tionality takes place.
no down time?);
• Monitoring of the service
(What aspects need moni- User stories
toring? How do we update User stories are a fantastic way
our monitoring with each to capture product requirements
change?); from the perspective of the ex-
pected outcome. User stories
• logging (What information
have helped many developers
should be logged? Why? And
(ourselves included) to think
in what style?);
about problems from the end us-
• alerting (Who? When? How? er’s point of view and to focus on
Why?), solutions to problems rather than
• the testability of the service; simply following instructions.
We’re referring, of course, to the
• security and compliance as- way that user stories focus on
pects such as encryption the “what” rather than the “how”
models, data protection, PCI (a good user story presents the
compliance, data legislation, problem and leaves the solution
etc.; and up to the developers).
• operational performance.
User stories are often written in
As one of us (Matthew Skelton) this format:
has written:To avoid “building in
legacy” from the start, we need “As a___,

well aware that they can wreak
havoc with our sprint commit-
ments. Two-week sprints seem
to add a lot of value in terms of
helping people to focus on a real-
istic target, but an unpredictable
environment can make it hard to
determine exactly how much of
our backlog we can achieve —
difficult, but not impossible.
If we measure the average

amount of work we can burn
through from our backlog, and
the average amount of interrup-
tions we get from the production
platform, we can essentially de-
duce two velocities.
Figure 3
Our backlog velocity is the rate at
which we can complete planned
I want___ s Sprints work from the product/service
Two-week sprints feel about right
backlog while the unplanned
o that___.” for developing new features,
velocity is the capacity to handle
testing and deploying, and then
work that hits the team during
This forces us to write them from demonstrating them to stake-
the sprint. Tracking these two
a user’s perspective (although holders. Any longer and it would
velocities allows us to plan more
not necessarily an end user). be hard to maintain focus, as well
effectively.
as pushing out the feedback loop
However, over the years, we have to a slightly uncomfortable de-
Kanban is of course another op-
found that writing operability lay. Any shorter — say, one week
tion, which can accommodate
stories using this format doesn’t — and suddenly meetings and
both planned and unplanned
really offer the same benefit. This other ceremonies take up an in-
work, and is often the framework
may be because the user per- ordinate percentage of our sprint
of choice for teams who have
spective has no impact on the time, meaning the amount of
little insight into what they’ll be
way the solution is technically work we can get done feels tiny.
working on in a week’s time. It
implemented. Regardless, it feels So two weeks feels right for many
can also be highly effectively to
a little redundant writing “As a people. It’s just the right amount
deliver longer-term projects/re-
sysadmin” or “As a developer” if of time to get your head down
leases, but requires high levels
we’re implementing the solution and focus on what you’re com-
of discipline in ensuring that the
ourselves. mitted to doing.
backlog is continually and cor-
rectly prioritised.
It’s not particularly unusual to see This is great if you’re developing
technical backlog items written in a new product, but what if you’re
a backlog without adopting the iterating through some improve- Sprint Planning
“As a___, I want___ so that___” ments or developing the next If we’re doing sprints, then we’ll
format, and similarly we tend not version of your product? need to do sprint planning. To
to recommend the format for bring a DevOps perspective to
operability features. We instead Who’s going to look after all the our sprint planning, we need to
prefer to use a “what and why” constant issues that arise from do the following:
format, which simply lists what the production platform?
needs doing and why (to provide • Invite ops/infrastructure/sup-
context). (Figure 3) If we’re exposed to frequent inter- port people to the planning
ruptions such as these (or equally session.
damagingly, varying degrees of
such interruptions) then we’re

We have to take a • Discuss not just product func-
tionality, but operability fea-
but also to software delivery and
maintenance.
tures as well.
fresh look at some • Plan their place in the upcom-
Another option is to transition the
role to an agile coach, who ad-
ing sprint.
well-established • Take into consideration time
heres to the principles and values
of agile in a way that’s sympathet-
and effort that will be con- ic to our new processes and not
concepts within Agile, sumed by interruptions
— that is, unplanned work
constrained by the prescriptive
rules of the Scrum framework.
such as the skillsets coming from the production

platform such as bug fixes, es-
The last thing we want is a Scrum-
Master who doesn’t appreciate
calations, etc. This value is our the purpose of DevOps; that’s
and roles within a unplanned velocity and effec-
tively acts to reduce our back-
simply going to create an even
bigger divide between the devel-
Product Team, the log velocity. The higher our
unplanned velocity, the lower
opment and operations sides.
our backlog velocity will be.

Product Backlog itself, Product owner
In our blended agile and DevOps
Definition of done
and how we plan and A popular definition of done is
“passed UAT”, which is basically
environment, our product owner
needs to understand the impor-
tance of operability more than
execute iterations. another way of saying “the busi-
ness has signed off the feature”. anyone.
But this largely forgets about op-
erability, security, performance, In SaaS, PaaS, and serverless en-
and so on. For a story to be con- vironments, a lot of the value is
sidered done, it needs to be ready hidden — it’s not in the front end.
to go live (or better yet, be in the The value is in how our services
live environment already). This work. It could result in saving
means it needs to be scalable, time, saving money, increased
functional, monitored, secure, performance, reduced risk, im-
and obviously deployable! If our proved reliability, or any other
story doesn’t satisfy all of these, hidden value. Product owners
it’s not done. now need to get this, because
ultimately they’re responsible for
guiding the priorities.
ScrumMaster
Bearing in mind that we’re going
to need to bend or break some Continuous integration
existing Scrum rules (see above (CI) and continuous
for examples), the role of the delivery (CD)
ScrumMaster is thrown into ques- Some people recommend sep-
tion. Even if we want to maintain arating CI and CD tooling, pre-
a process that closely resembles sumably because while CI is more
Scrum, the fact is it isn’t Scrum; dev-focused, CD has a more holis-
it’s going to be a blend of Scrum tic view.
and DevOps.
Whichever way we look at it, CI
Certain aspects of the ScrumMas- and CD are more than just tools;
ter role are still perfectly valid, they’re actual ways of working.
such as removing impediments, There are big differences be-
but the ScrumMaster will now tween having a CI system and do-
need to remove impediments ing continuous integration. The
not just to software development same can be said of CD.

Figure 4
In our DevOps/agile blended Conclusion principles right at the beginning

environment, it’s essential that The most popular agile frame- of our development process, be-
we not only use CD as a delivery work, Scrum, was designed for a cause bolting on a bit of deploy-
mechanism but also as a guiding time when teams tended not to ment automation at the end isn’t
set of principles and practices. worry about operational issues going to help us build more scal-
This matters because CD brings such as scalability, deployability, able, deployable, and manage-
development and operations monitoring, and maintenance. able solutions.
into the same frame. A good CD
pipeline like the one below will As a result, practices within Scrum We need to consider DevOps
visualise all of the important (and other agile frameworks) right from the moment we hire
steps in getting software deliv- overwhelmingly focus on what our team members, through the
ered successfully and regularly — we might loosely define as the planning and building of our
we can see how important it is to development aspects of software products, right through to their
have available test infrastructure, delivery, and less focused on the ultimate retirement.
reliable testing frameworks, good operational aspects.
monitoring, and deployment au- This means we have to take a
tomation. (Figure 4) DevOps helps to redress that im- fresh look at some well-estab-
balance, but has little influence lished concepts within agile, such
Remember the eight principles over the practices that happen as the skillsets and roles within a
and four practices of CD, as out- during the development phase it- product team, the product back-
lined by Dave Farley, paying par- self. The lack of definition around log itself, and how we plan and
ticular attention to the key prac- DevOps and the lack of a pre- execute iterations.
tices of “build binaries only once”, scriptive framework mean that
“use precisely the same mecha- there’s little or no information on Many teams have successfully
nism to deploy to every environ- how to bring DevOps thinking adapted their agile practices to
ment”, and “if anything fails, stop into our agile software-develop- become more DevOps aligned,
the line!” — but above all, heed ment processes. but there’s no one-size-fits-all
this message: “Everybody has re- solution available, just a collec-
sponsibility for the release pro- To maximise the value of agile tion of good patterns.
cess.” and DevOps, we must start to
implement some of the DevOps

DESIGNING DELIVERY
BOOK REVIEW AND INTERVIEW
by Manuel Pais
Jeff Sussna’s book, Designing Delivery, not only

dives into how organizations should think about
their business approaches but also shows deep
understanding of the expectations of customers
today. These include speed of delivery of new
features (or bug fixes), operational excellence
(access anytime, anywhere) and active brand
engagement (across multiple platforms, device)
during the entire customer lifecycle.

In short, the book puts today’s challenges as a service company (and most businesses have become one by
now) into a coherent narrative and a comprehensive structure of capabilities required to succeed. Although it
clearly targets an audience of C-level executives (as they should be able to see or have access to their organi-
zations’ global picture and its shortcomings), the executives should be wary that attempting to become a “cus-
tomer-centric brand” that continuously redesigns itself requires pre-existing levels of maturity. If development
teams are not agile, operations don’t embrace DevOps, or design teams don’t value design thinking, then they
should start there.
For everyone else, this book is an interesting read and a prediction of the kinds of skills and attitudes most ser-
vice organizations will come to demand from their people.
Part I of the book explains why service delivery must be a customer-centric process: never finished, continuously
adapting to customer needs. A paradigm shift is required for roadmap-driven organizations that optimize inter-
nal delivery and management processes. They instead need to focus on seeing their product value from their
customers’ perspectives, and realize how important it is to listen and act swiftly on customer feedback.
Part II breaks down the redefinition of quality under this new business modus operandi. It frames the expecta-
tions of customers (often unaware themselves) in four dimensions: customer outcomes (jobs to be done), access
(anytime and anywhere needed), coherency (across time and multiple customer touch points), and continuity
(adaptation to evolving customer needs). QA’s new role needs to include not only validation of the customer
journey but also the internal journeys of the other roles involved in service design such as development, oper-
ations, customer support, etc.:
“QA is about validating a service’s ability to harness change and helping them continuously improve.”
Part III introduces promise theory as a basis for thinking of services as chains of promises made to customers.
Inherent in a promise is the possibility of failing to fulfil it. Thus, a service organization thinking in promises is
preparing and embracing failure, as long as it leads to active learning which in turn can lead to more successful
promises. Tightly linked to this view is the understanding that all socio-technical systems we use today are com-
plex in nature, and thus fail in unpredictable ways. Sussna writes:
“Asking whether a service is making the right promises naturally maps to validating requirements. Asking what
a service needs to do to keep its promises naturally maps to identifying implementation holes and bugs. Asking
what other promises a service needs naturally maps to integration testing.”
InfoQ reached out to Sussna in order to better understand some of the ideas behind this book.
InfoQ: What was your motiva- gether in my mind; the book was to data-center automation plat-
tion to write this book? an attempt to coherently com- forms. That background gives
municate a new way of thinking me a somewhat unique perspec-
Jeff Sussna: I’d been giving a
along with a method for apply- tive on IT in general and DevOps
series of talks at various confer-
ing that new thinking. in particular.
ences over the course of a couple
of years. All my talks were essen-
I also have a liberal arts back-
tially about the same thing: the
ground. I grew up looking at
need to integrate design with InfoQ: How did your profes- pictures of Frank Lloyd Wright
engineering in today’s IT organi- sional experience influence its buildings; in college, I studied
zations. During the same period, content? anthropology, sociology, and
I read a biography of Norbert
Sussna: I’ve built systems and art theory. A few years ago, I had
Wiener and immersed myself in
led organizations across the an epiphany when I read Tim
the history and theory of cyber-
entire development/QA/opera- Brown’s book on design think-
netics. I also encountered Mark
tions spectrum. I’ve worked on ing, Change By Design. I sud-
Burgess’s work on promise the-
everything from compilers to denly understood that while I
ory, which is very cybernetic at
content-management systems was not a designer in the formal,
heart. These themes all came to-

traditional sense, I approached fined by how well and quickly themselves as providing service
problems from a design-thinking they adapt the services they to each other, on an ongoing ba-
perspective. From there, I dis- provide to ever-changing sis, with a greater shared value
covered the design-thinking off- customer needs. Is it fair to say (serving the customer) in mind.
shoot called service design. It oc- then that service delivery must
curred to me that, since IT is more encompass multiple areas, not
and more about service and more only the technical delivery (IT)
and more directly relevant to or- area? InfoQ: Your ideal cocktail for
dinary people, we need to design continuous evolution of digital
Sussna: Absolutely! Customers services includes not only
it as we would any other service.
judge service value by the en- cybernetics, agile, and DevOps
tirety of their relationship with but also design thinking.
In sum, that’s a long-winded way
the provider. Enjoying a restau- However, it seems that design
of saying that my background,
rant requires not just good food work is still disconnected from
like the content of the book, is
and not even just good food and the technical delivery work. Do
very interdisciplinary.
service, but also good ambiance, you agree and how would you
price, parking availability, sur- improve those connections?
rounding neighborhood, and a
decent website for looking up the Sussna: Integrating design with
InfoQ: You mention that
menu. As all business becomes engineering isn’t just about get-
cybernetics is a concept that
service business, and all business ting designers and developers
precedes the sci-fi image most
becomes digital business, the to work more closely together.
of us have of it. Would you like
boundaries between IT and the It’s also about applying design to
to explain why the original
rest of the organization begin to IT itself, and approaching IT as a
meaning is important?
dissolve. If your functionality is design act. How do we reimagine,
Sussna: At its heart, cybernetics great but it doesn’t scale, or isn’t for example, ITIL incident man-
views control as circular rather secure, or is hard to on-board, or agement, and helpdesk software
than linear. Even at the simplest your customer support is poor, or and workflows, in user-centered,
level, does the thermostat con- the website that explains what service-centric ways? Do we fo-
trol the temperature of the air in your service actually does and cus on closing tickets or on help-
the room or does the air control how/why to use it is incompre- ing users solve their problems?
the thermostat? The answer is hensible, or you handle outages
“yes”. As we move from a product or security breaches clumsily…
economy, which stressed push- — any and all of those defects
ing things and messages towards InfoQ: Could you briefly ex-
can degrade quality in your cus-
customers, to a service economy, plain what continuous design
tomers’ eyes.
which stresses companies and is for you and how it fits with
customers co-creating value to- continuous delivery and qual-
gether, we need to take a more ity?
circular approach to control. Also, InfoQ: What are the effects, at Sussna: Continuous design
as our world and the problems the organizational level, of this breaks down the boundaries be-
we face become more complex, idea of continuously evolving tween design and operations.
we need to think more in terms services? Do they require new Just as the thermostat continu-
of steering our way through com- organizational structures to be ously responds to changing air
plexity rather than engineering successful? temperature, so too the continu-
static solutions. The word “cy- ous-design organization continu-
Sussna: They definitely need
bernetics” comes from the Greek ously responds to feedback from
people and groups to work across
kybernetes, which literally means operations. Continuous design
disciplines in new ways. Personal-
“good steering”. Problem solving operates on multiple levels:
ly, I’m less concerned with explic-
in the face of complexity is less
it org charts than with behavior
about “Does it work?” and more 1. Continuous through time
and attitude. I think it’s perfectly
about “Are we steering well?” — unlike designing a coffee
fine for designers and developers
and testers to report to different mug that you hope to pro-
managers. What they do need to duce and sell for 40 years, the
do is to collaborate and empa- design process never ends.
InfoQ: The book stresses that
today’s organizations are de- thize with each other, and to see

2. Continuous throughout the to shift from stability, expressed attitude and set of behaviors that
lifecycle — A/B testing, game as mean time between failures, to lets that feedback flow through
days, and chaos monkeys im- resiliency/adaptability, or mean the organization, instead of be-
ply that design happens in time to repair. We are succeeding ing blocked by organizational
production as much as any- as long as we are listening and re- and behavioral silos.
where else. sponding to our customers.
3. Continuous throughout the

organization — each part of
the organization, whether InfoQ: You mention every or-
the design or development ganization is now part of a ser-
department, or the two-pizza vice ecosystem, where every-
microservice team, is a ser- one plays both producer and
vice provider to other parts consumer roles. What are some
of the organization, and thus good practices for managing
must continuously design your service dependencies?
and deliver its capabilities on Sussna: This is precisely where
its consumers’ behalf. promise theory comes in. The
use of the word “promise” means
4. Continuous delivery is a nec- first of all that we make commit-
essary but not sufficient comments, and work hard to honor
ponent of continuous design. them, regardless of the promises
You could say that continu- made to us. On the other hand, it
ous quality is a measure of means that we don’t assume the
the success with which an promises we rely on will be kept.
organization is continuously Whether we’re talking about hu-
designing. man promises (“I promise to help
you solve your problem when you
call customer support”) or system
promises (“I promise to serve
InfoQ: The quality of a finished
webpages in < 10 ms, regardless
software product used to be
of the number of users”), deliv-
characterized by the num-
ering service as promises creates
ber of defects found and, in
the loose coupling combined
business terms, the number of
with attractive force that lets co-
sales or licenses. How does the
herent systems emerge without
definition of quality change
causing large-system brittleness.
when thinking in terms of con-
tinuously running services?
Sussna: The reason we need to
approach digital business cyber- InfoQ: Do you believe custom-
netically is that we are actually er-centric brands require a new
continuously failing. Even if we approach to service support in
design and implement a new order to re-engage disgruntled
feature perfectly, as soon as the customers and channel their
customer starts using it, their feedback to the rest of the
needs begin to change. “This is organization?
great; now can you make it do X?” Sussna: I believe the entire ser-
is a common customer refrain. vice organization, from customer
The very act of using something support back, needs to shift their
changes the user, thus changing mindset from “making things for
the design problem. We are there- customers” to “helping customers
fore continuously narrowing and achieve desirable outcomes”. Per-
then discovering/creating new haps ironically, ITIL captures this
gaps between needs and capabil- definition of service perfectly. If
ities. For this reason, quality has we make that shift, we create an

As our world and
the problems we
face become more
complex, we need
to think more in
terms of steering
our way through
complexity rather
than engineering
static solutions.

PREVIOUS ISSUES
54
Reactive JavaScript
This eMag is meant to give an easy-going, yet varied

introduction to reactive programming with JavaScript.
Modern web frameworks and numerous libraries
have all embraced reactive programming. The rise in
immutability and functional reactive programming
have added to the discussion. It’s important for
modern JavaScript developers to know what’s going
on, even if they’re not using it themselves.
55 53
Serverless Computing
Cloud
Native
In this InfoQ eMag, we curated some of the best
In this eMag, the InfoQ team pulled together stories that serverless content into a single asset to give you a
best help you understand this cloud-native revolution, relevant, pragmatic look at this emerging space.
and what it takes to jump in. It features interviews with
industry experts, and articles on key topics like migration,
data, and security.
Microservices vs.
52
Monoliths - The Reality
Byond the Hype
This eMag includes articles written by experts who

have implemented successful, maintainable systems
across both microservices and monoliths.

Faster Smarter DevOps EMag 1512945600237

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Faster Smarter DevOps EMag 1512945600237

Uploaded by

Copyright:

Available Formats

FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT

ARTICLE Q&A BOOK REVIEW & INTERVIEW

How to Deal with Merging Designing

The Things I Learnt About DevOps

How to Deal with COTS Products in a DevOps World

Merging Agile and DevOps

Designing Delivery Book Review and Interview

facebook.com @InfoQ google.com linkedin.com

Derek Weeks John Clapham

Matthew Skelton Jeff Sussna

FASTER, SMARTER DEVOPS

Call it DevOps or not, if you are concerned about

Waterfall to agile, agile to continuous integration, continuous integration to continuous de-

6 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

However, because adopting DevOps tools is so easy, the trend is to focus

Let’s look at an example of application security. A number of technology

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 7

Also, one solution might be built to support asynchronous investigations

One approach I have used to establish a common language is to share a

8 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Pipeline and business metrics

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 9

That said, you need to focus on

• Are you completing activities

10 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 11

KEY TAKEAWAYS THE THINGS I LEARNT ABOUT

The same ways of working are

It takes a conscious effort to This is a true story, based on a talk

12 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 13

own reaction to car, the heat has intensified, and

look upon them

14 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Months later, we are contacted by an

If it matters, share it — you

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 15

KEY TAKEAWAYS HOW TO DEAL WITH

COTS products will continue to

Version control for COTS

16 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 17

I would love for vendors to reach

18 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Figure 1: A custom IDE to intercept any code changes.

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 19

20 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Step 3: Support intelligent merges

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 21

The most practical way to deal with configura-

Some creativity is required and the bar is a bit

In one project, we were able to reduce non-val-

You can all move forward on your DevOps

22 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 23

KEY TAKEAWAYS MERGING AGILE

24 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 25

26 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

Faster, Smarter DevOps // eMag Issue 56 - Dec 2017 27

the moment we hire like 90% traditional features that

through the planning mance related or something to

28 Faster, Smarter DevOps // eMag Issue 56 - Dec 2017

If we measure the average