DevSecOps

The InfoQ eMag / Issue #70 / April 2019

IN PRACTICE
The Three Faces
of DevSecOps
DevSecOps is here to stay, as
more vendors use the term.
But what is it? A security
solution that supports DevOps
technologies, or adapts to
DevOps methodologies,
or embraces the DevOps
philosophy?

Five Lessons Security

Can Learn from DevOps
New approaches in security
are needed to address the
challenges of DevOps. They
must incorporate practices that
rely on modularity, automation,
standardization, auditability, and
mirrored systems.

A Security Approach
for a Cloudy World
Does your approach to
application and data center
security change when adopting
cloud services? InfoQ reached
out to Pete Cheslock, from
Threat Stack to learn more.

FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL SOFTWARE DEVELOPMENT

 '-' WhiteSource

Automate Security
Into Your DevOps Pipeline

·r,f
a r
11·
a

FIND OUT HOW

InfoQ @ InfoQ InfoQ InfoQ

DevSecOps
IN PRACTICE
IN THIS ISSUE

The Three Faces of

DevSecOps 08 Five Lessons Security Can Learn
from DevOps 14
A Security Approach for a Four Steps to Continuous
Cloudy World - interview
with Pete Cheslock 18 Security in Container
Deployment 22
Developing a Secure and
Scalable Web Ecosystem
at LinkedIn 27

GENERAL FEEDBACK feedback@infoq.com / ADVERTISING sales@infoq.com / EDITORIAL editors@infoq.com


Manuel Pais
is a DevOps and Delivery Consultant,
focused on teams and flow.
Manuel helps organizations adopt
test automation and continuous
delivery, as well as understand
DevOps from both technical and
human perspectives. Co-curator of
DevOpsTopologies.com. DevOps
lead editor for InfoQ. Co-founder of
DevOps Lisbon meetup. Coauthor
of the book “Team Topologies:
Organizing Business and Technology
Teams for Fast Flow”.
Tweets @manupaisable
A LETTER FROM
THE EDITOR
DevSecOps is now clearly es-
tablished as an essential ap-
proach to build in security early
in an application’s lifecycle. The
necessary tooling is also avail-
able to integrate security activi-
ties in modern software delivery
pipelines. Yet we keep hearing
about high profile breaches in
the news (from which you might
or might not want to think about
how many lower profile breaches
happen on a regular basis). Why
are the tools not enough? With
this eMag, we present you expert
security advice on how to effec-
tively integrate security practices
and processes in the software
delivery lifecycle, so that every-
one from development to secu-
rity and operations understands
and contributes to the overall
security of the applications and
infrastructure.
The first question we need to ask
is what does DevSecOps mean
for our organization, our context,
our market (including applicable
regulations) and our current team
structures and interactions? Guy
Podjarny’s article gets us asking
the right questions on what is our
interpretation of DevSecOps: is
it just about automating current
security testing, bringing it in
our delivery pipelines with the
usage of the necessary tools? Or
is it actually about changing our
approach to security controls and
revisiting when and how those
activities are performed in the
overall application lifecycle? Or
do we need to re-consider who
is really responsible (and pos-
sibly re-think our team struc-
tures and skills) for ensuring our
applications, infrastructure and
networks are secure? There is
no single right answer of course,
but plenty of food for thought in
Podjarny’s article.
Wei Lien Dang’s article sheds fur-
ther light on how DevOps practic-
es like standardized and trace-
able configurations, immutable
infrastructure, blue-green de-
ployments and so on can help
security professionals reduce toil
and increase the speed at which
security controls are carried out
during the lifecycle, as well as
reduce the time it takes to de-
tect security issues at runtime.
Dang also points out the need
for security tools themselves
to follow modern architectural
patterns like modularity and ease
of integration.
In an insightful interview with rity concerns, approaches and
Pete Cheslock, you will find in- techniques for each phase.
valuable lessons learned on how
security changes when embark- Last, but surely not least, read
ing on a cloud migration journey about how LinkedIn kept security
by someone who has been there at the forefront while architect-
and done that. Cheslock points ing their platform’s scalability to
out emerging security risks support a 20x increase in mem-
around sprawling shadow IT in bers in a 6-year timeframe! All
the cloud or assuming that open while increase the frequency of
source code is secure because delivery from hands-on heavy
it’s openly available. But he also monthly releases to releases 3
makes the case that cloud ser- times per day.
vices can help us better under-
stand our systems at runtime in We hope this eMag can provide
the cloud and automate some an overview of concrete prac-
security practices with a lot less tices and help you get started
effort than on-prem. with the highest priority security
gaps in your organization today.
Containers have revolutionized Or simply clarify and help accel-
the way we package and deploy erate your current DevSecOps
applications and tools. They also initiative.
brought about intense discus-
sions and important changes in
how we secure both the infra-
structure where they run and
the applications inside them.
Fei Huang gives us a detailed
framework on building security
in containers in a continuous
fashion. He explains the impor-
tance of planning in different
phases (build, ship, preparation
and production) and provides
step-by-step guidance on secu-
CONTRIBUTORS
Wei Lien Dang
is VP of product at StackRox, a security
company building a new platform to protect
modern enterprise applications. Previously,
he was head of product at CoreOS and
held senior product-management roles for
security and cloud infrastructure products
at Amazon Web Services, Splunk, and
Bracket Computing.
Guy Podjarny
(@guypod) is Snyk’s co-founder and CEO,
focusing on using open source and staying
secure. Guy was previously CTO at Akamai
following their acquisition of his startup,
Blaze.io, and worked on the first web app
firewall & security code analyzer. Guy is a
frequent conference speaker & author of
O’Reilly “Securing Open Source Libraries”,
“Responsive & Fast” and “High Performance
Images”.
Fei Huang
is the co-founder and CEO of NeuVector, a
Docker-container network-security solution
that uses behavioral learning to secure
containers during run-time. Fei has over 20
years of experience in enterprise security,
virtualization, cloud, and embedded
software. He was part of the founding team
of Cloudvolumes (acquired by VMware)
and co-founder of Provilla, a DLP security
company (acquired by TrendMicro).
He holds several patents in security,
virtualization, and software architecture.
Richard Seroter
is the VP of Product Marketing at Pivotal,
with a master’s degree in Engineering from
the University of Colorado. He’s also an 11-
time Microsoft MVP, trainer for developer-
centric training company Pluralsight,
speaker, the lead InfoQ editor for cloud
computing, and author of multiple books on
application integration strategies. Richard
maintains a regularly updated blog on
topics of architecture and solution design
and can be found on Twitter as @rseroter.
James Baker
is a senior software engineer on LinkedIn’s
Feed (homepage) team, striving to
provide an excellent experience for
millions of LinkedIn members across
the world. Baker has been active in web
engineering roles since 2009 and has
been in some type of IT role since 2004.
He is based out of Silicon Valley and is
passionate about writing enterprise-level,
highly performant, secure, and accessible
web applications.
Mira Thambireddy
is an information security engineer at
LinkedIn, where she is a part of LinkedIn’s
Application Security and Penetration
Testing team. Previously, Thambireddy
worked as a security consultant in Silicon
Valley. She holds a master’s degree in
information security from Carnegie Mellon
University.
 The Three Faces of
The InfoQ eMag / Issue #70 / April 2019

DevSecOps
by Guy Podjarny, cofounder at Snyk.io

DevSecOps is here to DevSecOps is a hot buzzword
stay, as more vendors these days, and I love it – as well
as hate it.
use the term. But
what is it? A security I love it because it embodies a
solution that supports goal the app sec industry, my-
DevOps technologies, self included, has been trying
or adapts to DevOps to accomplish for years – get-
methodologies, or ting developers to own security.
embraces the DevOps This simple buzzword gives our
philosophy? mission a name and helps its
momentum, driving it towards
becoming the norm.
I hate it because, like all buzz-
words, it doesn’t properly reflect
the high complexity behind this
journey. Security is a broad topic,
spanning infrastructure, applica-
tion code, network stacks, third
party vendors, and, of course,
people. Changing how we apply-
security involves a lot of detail,
but buzzwords don’t have room
for such nuance. Inevitably, dif-
ferent people use it for different
meanings – or at least empha-
size different parts.

8

This isn’t surprising, and many
buzzwords have a similar effect.
More specifically, if you get three
experts in a room you’ll get 4
different interpretations of the
term DevOps, and 5 of the term
Cloud. While we’ll never get to a
single definition, writing down the
different points of view still helps,
as they give us a way to classify
what someone means when they
use the term.
Looking at its different uses, I see
three primary meanings attribut-
ed to the term DevSecOps. These
views represent different mile-
stones in the DevSecOps journey
and align to common interpre-
tation of the term DevOps itself.
Let’s go through each and dig
one layer deeper into this won-
derful and horrible term.
DevSecOps as “Security for
DevOps Technologies”
Perhaps the most literal transla-
tion of DevSecOps is to describe
security solutions for the wave
of technologies DevOps brought
along. Technologies like cloud,
containers and serverless, to
name a few, are key players in
the DevOps movement. Each of
these offers new functionality
and introduces new technology
stacks, which in turn have specif-
ic security needs.
The first step in supporting these
technologies is taking an existing
security product, which is used
primarily on older stacks, and
adding the technical components
necessary to use it in a DevOps
environment. These are mostly
minor adaptations of existing
tools - as opposed to substantial
change in the way the tool oper-
ates - simply meant to support
the new surroundings.
Let’s take container security as
an example. Symantec, McA-
fee, TrendMicro and others offer
endpoint security solutions
which include anti-malware,
monitoring for malicious network
activity and more. These mature
products have long worked for
servers and virtual machines,
and conceptually apply to con-
tainers just as much. However,
the division of controls between
the container and the machine
hosting it, along with various
other technical differences,
prevented these products from
The InfoQ eMag / Issue #70 / April 2019
supporting containers out of the
box and requiring them to adapt.
A good example of that is Trend-
Micro’s Deep Security product,
which added container support
in version 10 with a new split of
responsibilities between the the
host machine and the contain-
er. This change was enough for
them to claim participation in the
DevSecOps movement, although
the product is still primarily used
in traditional security context.
One step further down the path
of securing DevOps technolo-
gies are security solutions that
address the newsecurity needs
these technologies introduce.
Examples here include Cloud-
Checkr and Evident.io inspect-
ing cloud configurations to find
storage buckets inadvertently left
publicly accessible (and other
issues, as can be seen in the
figure below), or Snyk looking for
vulnerabilities in open source li-
braries. Handling newer technol-
ogies requires novel thinking, and
is often harder for incumbents
who are already predisposed
and structured to older ways of
thinking. It is therefore typically
the domain of startups, with big
9
The InfoQ eMag / Issue #70 / April 2019
Figure 1 / CloudCheckr’s cloud security assessment results
companies stepping in via acqui-
sitions as opposed to building a
solution from scratch.
Containers exemplify this sce-
nario well. The fact the isola-
tion between the host and the
container isn’t perfect created a
new risk - having malicious code
running in a container break out
of the container sandbox and
impact the host machine. Since
the same physical host often
runs containers at different levels
of sensitivity or even belonging
to different owners, sandbox
escaping is a real and immediate
threat. This and additional new
risks introduced by containers
opened the door to startups
focusing specifically on container
security, such as Sysdig, Aqua
and Twistlock, who harden the
container and flag attempts to
break out of it. These startups
typically identify themselves as
DevSecOps companies through
and through.
10
In both cases, the term
DevSecOps refers to securing
DevOps technologies. That said,
some of these solutions natu-
rally bleed into the way we ap-
plyDevOps too, which gets us to
the next meaning…
DevSecOps as “Security for
DevOps Methodologies”
Beyond its technologies, DevOps
drove adoption of powerful new
methodologies, such as contin-
uous delivery (CD) and micros-
ervices (followed by serverless).
These techniques lead to faster
development and more granu-
lar components, both of which
quickly break existing security
methodologies.
Once again, these new approach-
es require an evolution for exist-
ing players. Let’s look at CI/CD as
an example.
The adoption of continuous
delivery means “stopping for an
audit” in the midst of the release
process is no longer acceptable,
requiring strong automated secu-
rity tests in the pipeline instead.
This demand for automation
drew attention to static applica-
tion security testing(SAST) tools,
but these took too long – often
hours – to complete a single
code scan, which isn’t viable in
a frequent build environment. To
adapt, most SAST tools intro-
duced a concept of incremental
scans, only testing the code that
changed. These scans complete
faster and can therefore fit inside
the pipeline. Once again, these
tweaks are often featured as a
demonstration of DevSecOps
prowess.
While such adaptations are im-
portant, they’re often not enough.
Some new methodologies don’t
just change the technical setup,
but also break core concepts
existing tools rely on, requiring a
rethinking of the entire product
– again a realm where startups
excel. Microservices security
monitoring offers a good demon-
stration of this.
Moving from a monolith appli-
cation to microservices changes
the definition of an app. Instead
of monitoring one entity with a
set of inputs and outputs, data
now flows between multiple – at
times dozens or even hundreds –
of different services. Successful-
ly identifying attacks in real time
in such an environment requires
an entirely different security
monitoring solution, which scales
to monitoring a large number of
services and the data flows be-
tween them. Startups like Apore-
to tackle this problem by tracking
service to service communica-
tion and flagging unauthorized
connection attempts, while
startups like SignalSciences fo-
cus aligning per-service security
insights with their respective ops
dashboards.
Figure 2 / Aporeto maps microservice and flags untrusted
connections (Aporeto demo webinar)
Beyond overcoming the chal-
The InfoQ eMag / Issue #70 / April 2019
lenges of these new approaches,
such dedicated solutions also
capitalize on the opportunities
they present. For instance, while
containers are logically similar
to VMs, they are often stateless
and deployed in highly elastic
environments. As a result, it’s en-
tirely reasonable to shut down a
container which is behaving sus-
piciously and may be compro-
mised, as no data will be lost, and
the system will simply spin up a
new one. Such a harsh response
is clearly not an option for bare
metal hosts and is not viable in a
heavy virtual machine either. And,
indeed, you’ll be hard pressed to
find an endpoint security solution
that promotes destroying a ma-
chine on alert, while practically all
the container and microservice
security startups heavily promote
doing so.
Adapting to DevOps methodol-
ogies is a substantially broader
11
 interpretation of DevSecOps than
The InfoQ eMag / Issue #70 / April 2019
focusing on technologies alone,
and far more valuable in the long
run. It typically requires some
technology support as well (e.g.
you can’t secure microservices
without supporting containers)
but goes on to understand and
adapt security to the new pro-
cesses being enacted. And yet,
while it touches two parts of
the golden triangle (People, Pro-
cess, Technology) - it still over-
looks the most important one.  team should spend most of their
DevSecOps as “Security for
DevOps’ Shared Ownership
Philosophy”
DevOps isn’t – at its core - about
technologies or methodologies,
but about people and collabo-
ration. It’s about accepting that
keeping our software running
well is everybody’s problem, and
that we share the responsibility
for making it happen. DevOps
goes against the practice of devs
“throwing code over the wall” for
ops to handle and helps ops ap-
preciate the constraints develop-
ers face on a daily basis.
This cultural and philosophical
shift drives the new methodolo-
gies and technologies. It leads to
developers writing more operable
code, ops treating infrastructure
as code, and the horde of ensu-
ing software and techniques to
make it scale. This shift is what
truly makes business move fast-
er and compete better.
The last and broadest view of
DevSecOps follows the same
12
footsteps. To truly address secu-
rity at the DevOps pace, we must
embed it into the regular devel-
opment and operation processes.
Since security teams - like ops
teams before them - are vastly
outnumbered by developers, the
only way to achieve shared own-
ership is by having the bulk of the
daily security activity performed
by the development teams. The
remaining work should be done
primarily by ops, and the security
time empowering those teams,
governing and automating this
activity (“security as code” to
follow “infrastructure as code”).
For existing security solutions,
this is a tough change. These
businesses are built to cater to
security professionals, and their
entire go-to-market models,
as well as their world view, is
focused on that industry. They
sponsor security events, use the
terminology of security people,
and price their product per the
security industry’s standards.
The bigger the security vendor,
the more they’ll have to lose from
changing their target market,
Figure 3 / Chrome Dev Tools flags
vulnerable JavaScript libraries
thus the less likely they are to do
so.
More importantly, security prod-
ucts are designed primarily to
help the InfoSec person do their
daily job, and developer integra-
tions are often done from that
skewed perspective. The results
are similar to what would happen
if VMWare built an IDE, or Citrix
created a CI offering. Those are
powerful ops tools, but they don’t
think developer, and are unlikely
to design the right tool for that
audience.
Interestingly, this DevSecOps
philosophy fits the opposite
direction better, that is having
developer tooling companies
embrace security. These compa-
nies have already won the hearts
and minds of developers and can
now take those developers a step
further in their DevOps journey.
Source code vendors can tackle
code security, APM vendors can
monitor for security flaws, and
pipeline vendors can harden what
goes through them.
This is no longer a theoretical op-

The InfoQ eMag / Issue #70 / April 2019

portunity. In 2017, we saw GitHub
add security alerts to its repos,
Google Chrome flag vulnerable
JavaScript libraries in its built-in
dev tools, and Sysdig launcha
container security product. These
companies also have a learning
curve to understand how to build
security solutions, but they are
anchored in the right place.

Lastly, for startups, this is a gold-

en opportunity. Businesses that
sell to developers are far more
capital efficient than their secu-
rity counterpart, as the former
have far lower sales costs. In
addition, CSOs today are increas-
ingly bombarded with cold calls
they learn to ignore, but security
solutions already embraced by
dev teams will still capture their
attention. We now know devel-
opers are the new kingmakers in
the operations world, and secu-
rity is likely to follow the same
footsteps.
Right now, there are very few
security companies focusing
on developers. One good ex-
ample is in the authentication
space, where Auth0 is winning
developers over with great doc-
umentation, a bottom up pricing
model and strong community
forums and programs. Their
latest $55M Series D is one
indication of the payoff. Another
example I can give here is my
own company, Snyk, which fixes
vulnerabilities in open source de-
pendencies. We focused on de-
velopers throughout, from deep
Figure 4 / Bringing automated remediation into
the developer context inside GitHub
dev tools integrations, through
many thought leadership activi-
ties, to investing in automated re-
mediation which helps develop-
ers more than auditors, and it’s
paying great dividends.
Stats show that organizations
that truly adopt the DevOps
philosophy get better business
results, regardless of the tech
and methodologies they used. I
expect organizations that truly
adopt this DevSecOps interpre-
tation to get the equivalent boost
in keeping themselves and their
users data secure.
Understanding DevSecOps
DevSecOps is a hot buzzword,
and it’s not going away. The term
is quickly entering the InfoSec
buzzword bingo, and the hotter
it is, the more companies and
vendors will use it, regardless of
what it means.
The next time you come across
a solution using this term, I hope
this article will help you classify it
properly. Is it a security solution
that supports DevOps technolo-
gies, adapts to DevOps method-
ologies, or embraces the DevOps
philosophy and helps you change
your organization? Or perhaps
it fits multiple buckets? With the
right classification, you can help
remove the noise, and focus on
the DevSecOps mission.

13
The InfoQ eMag / Issue #70 / April 2019

New approaches in
security are needed to
address the challenges
of DevOps. They
Five Lessons must incorporate
practices that rely on

Security Can Learn modularity, automation,

standardization,

from DevOps
auditability, and
mirrored systems.

by Wei Lien Dang, VP of Product at StackRox

As DevOps becomes increasingly
crucial to the modern enterprise,
security professionals must ask
themselves what they can learn
from this culture shift. DevOps is
a blend of organizational ideas,
processes, and software tools
that has enabled some of the
world’s largest companies to
improve productivity, achieve
faster time to market, and de-
liver higher-quality products
that impact their bottom lines at
blazing speeds. Enterprises like
Netflix, Google, and Amazon are
examples of high-performing,
agile organizations that consid-
er DevOps to be foundational
to the success of their digital
businesses.
Companies with effective DevOps
practices understand that in
the technology-driven, winner-
take-all, competitive landscape,
a business’s success often
depends on its ability to contin-
uously learn, innovate, and more
quickly deliver cutting-edge, cus-
tomer-delighting products. At the
same time, this creates challeng-
es for other organizational func-
tions such as security. Delivering
distributed applications more
quickly and easily substantially
increases risk to businesses that
cannot secure them at the same
speed at which they are deployed
and scaled. Security workflows,
tools, and operations must
change to keep up.

14
More recently, the need to bal-
ance DevOps speed with ex-
isting security requirements
has resulted in a model called
DevSecOps. DevSecOps is based
on the principle that “everyone
is responsible for security.” It
emphasizes how application
developers can build security
checks into their integration
and deployment pipelines. But
DevSecOps has focused far less
on security during runtime, which
is when the business’s applica-
tions and data are most vulnera-
ble to attacks. Runtime security
encompasses all types of threats
once an application is running
and includes functions such as
attack detection, incident re-
sponse, and policy enforcement.
These functions still largely
rely on siloed tools and manual
workflows. Additionally, runtime
security cannot and should not
be the responsibility of everyone
throughout the enterprise; these
functions are best handled by
security professionals.
Besides building security into
DevOps workflows, security
teams should evaluate how they
can incorporate DevOps princi-
ples into their tools and process-
es. Here are five DevOps practic-
es that security professionals can
learn from.
Build modular systems
Key to the DevOps philosophy
is building systems that can be
more easily managed by small
teams. Many of the tools and
approaches used by DevOps
teams favor applications that are
The InfoQ eMag / Issue #70 / April 2019
assembled in modular fashion.
Some examples include micros-
ervices architectures, container
technologies such as Docker, and
12-factor application methodol-
ogy. Microservices let developers
focus on optimizing individual
application pieces that interact
more easily with other systems
via APIs. This streamlines de-
velopment, integration, and
deployment.
Security lessons: Security teams
must grapple with a patchwork
of dozens of security tools that
do not easily integrate. Bringing
modularity to security will dra-
matically reduce the operational
complexity, time, and costs of
integrating and managing these
solutions. Security professionals
should look for new tools that are
built using the same frameworks
that enable modular application
development. Security delivered
using microservices running in
Docker containers can be easily
distributed across entire clusters
of applications, orchestrated by
systems like Kubernetes, Meso-
sphere DCOS, or Docker Swarm.
This means security tools can
blend in as just another set of
applications that are automated
and orchestrated alongside the
applications they protect.
Depend on automation and
scalability
The overriding goal of DevOps
is to achieve greater speed and
agility to better serve customers.
Teams reach this goal through
15
 high levels of automation and
The InfoQ eMag / Issue #70 / April 2019
scalability that are designed to
make infrastructure fully pro-
grammable. For example, Netflix
has focused on automating the
company’s entire software-re-
lease platform so that it can
scale on demand to thousands of
servers in minutes, ensuring that
customers always have access
to their content. Automation has
the added benefit of reducing the
likelihood of manual operator
error, which is a frequent source
of costly service interruption or
downtime.
Security lessons: Security’s
mandate is to mitigate the risk
and impact of attacks on the
business. Achieving automation
enables faster time to detection,
which gives security teams more
time to optimize response and
recovery. As new applications
are continuously deployed and
scaled at the speed of the cloud,
attack surfaces can scale and
change just as constantly and
quickly. Thousands of micro-
services applications may be
launched or destroyed within the
span of a few seconds, leaving
gaps in visibility and data collec-
tion. Security tools must incorpo-
rate automation and scalability to
keep up with these new appli-
cation delivery models in order
to minimize potential attacks
while preserving existing devel-
oper toolchains. Automating the
runtime security lifecycle also
mitigates the likelihood of con-
figuration errors that attackers
can potentially exploit, helping to
16
reduce security incidents. Relying
on traditional, fragmented, manu-
al security processes is no longer
sufficient.
Use standardized configurations
In DevOps, teams take advantage
of consistent baseline config-
urations to identify operational
issues more easily. Teams pack-
age “infrastructure as code” in
the form of base images, applica-
tions, and dependencies needed
to run services. This ensures
that systems are identical and
reproducible and drives better
operational hygiene. Monitoring
for deviations against stan-
dardized configurations allows
operators to quickly identify and
troubleshoot issues. Treating
infrastructure as immutable, so
that running systems are nev-
er reconfigured, is increasingly
becoming a best practice that
is facilitated by the adoption of
application container technol-
ogies such as Docker. Running
systems are simply replaced with
new ones that incorporate any
desired changes. This approach
allows teams to easily stop prob-
lematic systems and relaunch
into known, good states at the
same time.
Security lessons: Security teams
can similarly implement ap-
proved security configurations,
fingerprinting, and profiling to
reveal potential issues. These
standardized configurations are
expected to conform to normal
system activity. Runtime activ-
ity that diverges from normal
settings can highlight anoma-
lous or malicious patterns that
reflect live attacks or indicators
of compromise. When detecting
such activity, security teams can
treat infrastructure as immutable
to quickly stop impacted sys-
tems and launch new, unaffected
ones without having to deal with
the complexity of patching live
systems.
Maintain an auditable “source of
truth”
DevOps culture helps foster
greater collaboration across
teams by using techniques
like revision tracking and ver-
sion control to maintain full
transparency, improve change
management, and streamline
system recovery. Modifications
to configurations are recorded to
capture details such as when the
change occurred, who requested
the change, and the impact of
the change. Using this approach,
development and operations gain
deeper visibility into the complete
lifecycle of both applications
and infrastructure, which helps
facilitate better cross-function-
al communications and greater
accountability.
Security lessons: Security teams
should ensure that they imple-
ment tools to collect compre-
hensive data from all systems to
maintain an actionable “source
of truth” that is readily accessi-
ble. Currently, most datasets are
either siloed or aggregated in a
centralized SIEM system without
the necessary modeling required
to make them actionable. Even
outside of use cases such as
governance and compliance that
require audit trails, capturing
every change to any rules, con-
figurations, and security policies
helps streamline collaboration
and accountability within securi-
ty teams. Security professionals,
who may each focus on specific
parts of the runtime security
lifecycle from configuring instru-
mentation to forensics, can more
easily take coordinated action
when security incidents arise.
Leverage mirrored, active-
standby deployments
Teams often use blue/green
deployments to minimize down-
time. This technique leverages
two identical production envi-
ronments; at any given time,
one is active and the second is
on standby. New releases are
only tested in the standby en-
vironment. Once teams verify
the changes, they activate the
standby environment and switch
the active environment to stand-
by. This allows teams to safely
roll back new releases if they
encounter unanticipated bugs or
operational issues without ever
having to make changes to any
live systems.
Security lessons: Security teams
can apply the same techniques
to patch management and
security updates. They can test
fixes on mirrored systems before
releasing them in active produc-
tion environments. For example,
some operating system vendors
now use dual active/passive
The InfoQ eMag / Issue #70 / April 2019
root partitions to enable simpli-
fied patch management with the
ability to safely roll back versions
if required. This has the added
benefit of increasing system
availability, which falls under
the confidentiality, integrity, and
availability (CIA) framework for
security management. Security
teams can take advantage of
this approach to first observe
the stability and efficacy of fixes
that are meant for production
environments. They can ensure
that security issues will be fully
remediated before implementing
them at large scale.
Conclusion
Businesses of all sizes are
seeking greater agility and speed
to fuel digital transformation.
By embracing DevOps, appli-
cation developers can focus on
what they do best: building and
shipping new software. Security
professionals should also con-
tinue to focus on what they do
best: keeping attackers out. Just
as DevOps emerged to meet new
business needs, new approach-
es in security are now needed
to address the challenges of a
DevOps-driven world. These
new security approaches them-
selves must incorporate DevOps
practices that rely on modularity,
automation, standardization,
auditability, and mirrored sys-
tems. By applying these five
DevOps principles to new tools
and workflows, security can help
businesses succeed.
17
The InfoQ eMag / Issue #70 / April 2019

INTERVIEW WITH PETE CHESLOCK

A Security Approach for
a Cloudy World
by Richard Seroter, VP of Product Marketing at Pivotal

Does your approach to application and data-

The Interviewee center security change when adopting cloud
services? To learn more about this topic, InfoQ
reached out to Pete Cheslock, head of operations
and support teams at Threat Stack.

Pete Cheslock
InfoQ: Have the on-demand, to computing resources their in-
As the head of Threat
self-service capabilities of cloud ternal technical teams are unable
Stack’s operations and
support teams, Pete IaaS forced a change an organi- to help them with. Now devs and
Cheslock is focused
on delivering the
zation’s security approach? If so, even ops teams can circumvent
highest level of service, how?  security policies and leave their
reliability, and customer
satisfaction to Threat security teams behind to clean
Stacks’ growing user Pete Cheslock: Absolutely. up the mess.
base. An industry
veteran with nearly 20 Before, security teams could tie
years’ experience in themselves into the procure- InfoQ: In past talks, you’ve
technical operations,
he understands ment process to ensure that all covered how companies build
the challenges and
issues faced by
systems and applications would security into their process from
security, development, meet standards before being the start. Does that practice look
and operations
professionals and how deployed and consumed by different if your company already
we can help. Prior to customers. But now, anyone at has a dedicated information se-
Threat Stack, Pete held
senior positions at the company can go and create curity team? 
Dyn and Sonian, where an account for a cloud provider
he built, managed,
and developed and start deploying their ser- Cheslock: In many ways, the
automation and release
engineering teams and
vices. Commonly referred to as steps to bring security into an
projects. “shadow IT” — we’ve seen count- existing DevOps process at a
less companies where they will small company can work at a
have dozens or hundreds of AWS big company as well. The goal is
accounts as a way to get access working with your security teams

18

to involve them in the tooling
and workflows your dev and ops
teams are already using (like
continuous integration and deliv-
ery). Another goal for companies
that don’t have a dedicated secu-
rity team, or have no idea where
to start, is to start small and not
attempt to boil the ocean.
InfoQ: What are areas where
technical staff take the cloud for
granted and assume that their
workloads are inherently secure?  InfoQ: Conversely, are there
Cheslock: Many times, people
think that using open source
means their code is secure. As-
suming that “many eyes makes
secure code”. That concept has
been proven wrong time and time
again, with many vulnerabilities
in core pieces of open-source
technology. Closed-sourced ven-
dors are no safer, as we’ve seen
many news reports of critical
vulnerabilities in closed-sourced
software and hardware as well.
Since no one is safe, leverage
MITRE and the National Vulner-
ability Database to help people
understand their risk, help them
prioritize their security updates,
and finally talk about the risk and
reward of nightly security up-
dates and what it means to your
business.
native cloud capabilities that
companies *should* use that re-
lieve them of a particular aspect
of security?  InfoQ: Should companies wean
Cheslock: The great news is that
cloud providers like AWS are
doing great things in the security
space to help their users under-
stand better what is going on.
If you are running on AWS, you
The InfoQ eMag / Issue #70 / April 2019
can get tools such as CloudTrail
to audit all the API calls on your
account and you can use AWS
Config in order to audit your
systems and ensure they meet
your compliance rules. Finally,
you leverage the EC2-VPC (which
is default for all new AWS cus-
tomers) to segment your sys-
tems behind private non-routable
networks and use network ACLs
to restrict access. In many cases,
the tools are there to be more se-
cure running in the cloud, users
just need to learn what they all
are.
themselves off perimeter security
as a dominant application securi-
ty strategy, and if so, how?
Cheslock: When moving to the
cloud, you can still attempt to
maintain a perimeter by routing
19
 all your traffic through your inter-
The InfoQ eMag / Issue #70 / April 2019
nal data centers, but eventually
you may move off your internal
data centers entirely so you have
to be prepared. The reality is
that many companies use end-
point security tools to track and
monitor their users’ laptops and
mobile devices. We need to take
the same tactic with our servers
as well. Every server running
code is an endpoint, and enabling
a continuous security-monitoring
tool can help you track and iden-
tify more than just zero-day ex-
ploits. They can help you identify
internal bad actors who might
be stealing intellectual property
using valid credentials. The risks
to a company come from inside
often as much as from remote
attackers.
InfoQ: What security aspects
should NOT be lifted and shift-
ed from on premises to a cloud
environment?
Cheslock: We mentioned this a
bit in the previous section, but
edge-based network-securi-
ty-monitoring tools. The chal-
lenge you can get into is in the
cloud every one of your systems
could be acting as an endpoint,
and the edge of the network
could be wider than you would
normally experience in a single
datacenter. The challenge comes
into play when companies go
multi-cloud — now you need to
deal with disparate networks
and providers that in many ways
don’t natively integrate.
20
InfoQ: Do your security recom-
mendations change if someone
moves up the stack from bare
metal or virtual cloud servers up
to PaaS or functions?  vices likely not thinking about
Cheslock: Providers such as
Heroku, Google Cloud Functions,
and AWS Lambda really make
the concept of securing your
systems more interesting when
you don’t have any servers to run
your code on. These are often
referred to as serverless — your
code executes inside a provider
on systems that you likely don’t
have any control over. In many
ways, this can help make you
more secure as you are reduc-
ing the number of endpoints
you need to secure. But in the
end, this pushes your security
challenges over to the provid-
ers themselves. AWS uses their
Identity and Access Management
(IAM), meaning you are now in
full control of providing access to
your functions. You need to en-
sure the security is as least-priv-
ilege as possible. Additionally,
your code needs to get to the
provider somehow, which means
you’ll be running systems that do
the continuous integration and
deployment — that is, where add-
ing in security testing and static
code-analysis tools at the build
and deployment side.
InfoQ: How does data security —
including access control, encryp-
tion at rest and in transit, change
data logging — look different in a
services world?  ing our security posture.
Cheslock: Often, when compa-
nies run inside their own data
centers on networks they own,
they will run many of their ser-
encryption. This assumes that
physical security can help them
beat this problem. When moving
to the cloud and shared infra-
structure, encrypting your data at
rest can be critically important to
ensure data is not leaking due to
bugs or errors with the provider.
Additionally, when running on
someone else’s infrastructure,
you need to ensure you are using
TLS to secure your services
internally, similarly to how we run
SSL/TLS on our external web-
sites. Many tools make this very
easy to do nowadays.
InfoQ: Is it a false choice to de-
cide between speed and safety?
Cheslock: In the past, we used to
have to choose between speed
of deployment and the safety of
deploying and managing that
code. That was the pre-DevOps
conversations. How can we both
move fast and stay highly avail-
able. Well, in many ways those
are now considered solved prob-
lems. We are now at the point of
asking the same questions in the
DevOps and security spaces. I
think using many of the tools and
technologies that enabled devs
and ops to work better together,
we can tie security into those
same processes and procedures
and move quickly while increas-
 
 SPONSORED ARTICLE

3 DevOps Security

The InfoQ eMag / Issue #70 / April 2019

Challenges & How to
Overcome Them
While organizations continue to adopt, expand, and perfect their DevOps game, security has become an
urgent issue. Malicious attacks on application layers are on the rise, and it seems like almost every day
brings news of yet another data breach in an organization. Enterprises are coming to realize that while
DevOps tools and processes helping them stay innovative within tight release timelines, the risks of slack
security remain real, immediate, and extremely costly. This puts DevOps outfits under pressure to imple-
ment stronger and smarter security measures, and adopt a set of secure DevOps practices.

Security: Too Slow to Join the
DevOps Squad?
Secure DevOps is a relatively new
approach, and up until recently,
not many people thought it was
a good fit. Even security experts
point out that incorporating
security people into the DevOps
cycle can be challenging.
Security expert Michele Chubir-
ka says in her blog that, “While
many security people have a
good understanding of how to
find application vulnerabilities
and exploit them, they often don’t
understand how software devel-
opment teams work, especially
in Agile/DevOps organizations.
This leads to inefficiencies and a
flawed program.”
This led to a situation where the
security team was essentially out
of the DevOps picture, unable to
have a real impact on the final
product in an efficient manner.
Secure DevOps: Where DevOps
and Security Can Play Nice
Over the past few years, more
and more enterprises and orga-
nizations are making a concerted
effort to shift security practic-
es left and implement security
throughout the DevOps cycle,
ensuring that it doesn’t impede
time to market.
According to recent DigiCert re-
search, organizations are already
invested in secure DevOps. Re-
search results show that 49% of
the organizations surveyed said
that they are in the process of
integrating security with DevOps
and that another 49% said that
they already completed their
DevOps security integration.
Ensuring a Smooth Ride: 3
DevOps Security Barriers & How
to Overcome Them
Change is never easy for an
organization, and managing a
secure DevOps cycle is quite a
transition on many levels. Teams
and experts need to adjust them-
selves to a new organizational
structure, new processes need
to be adopted, new skills need
to be developed, and new tools
need to be integrated. As compa-
nies continue to adopt a secure
DevOps approach, they should be
aware of the common challenges
management teams face as they
set out on this journey.
Here are a few barriers organiza-
tions face on the road to ensuring
an efficient and innovative secure
DevOps cycle:
#1 Teamwork Makes the Dream
Work
Adopting a secure DevOps
approach requires teams and ex-
perts that aren’t used to working
together to cooperate, creating
and maintaining a development
lifecycle that delivers quickly and
securely.
READ THE FULL ARTICLE

21
 Containers face security risks at every stage, from building to shipping to
The InfoQ eMag / Issue #70 / April 2019

the run-time production phases. Securing them requires a layered strategy

throughout the stack and the deployment process.

Four Steps to
Continuous Security in
Container Deployment
by Fei Huang, Co-Founder & CEO of NeuVector

Containers face numerous se- containers forced to eat up Apache Struts exploit used in
curity risks at every stage of the system resources to crash other Equifax breach
deployment process, from build- containers, and plenty others. Apache Struts is a widely used
ing to shipping to the run-time framework for creating web ap-
production phases. These threats Many of these attacks start with plications in Java. It was initially
make achieving continuous previously undetected vulnera- believed that attackers used a
security essential to the integrity bilities, which can lie dormant for newly published Struts vulnera-
and safety of containers. Howev- years. Attackers exploit these to bility, CVE-2017-9805, to access
er, often these security measures gain a foothold, and then expand the Equifax data in 2017 but an
are skipped or overlooked in the into other hosts or containers. Equifax announcement indicated
rush to get the CI/CD pipeline
flowing faster. Many businesses
are paying the price for imple-
menting poor security years ago.
Hopefully, these mistakes won’t
be repeated in the migration to a
containerized environment.

Risks to containers and their

hosts come in many forms: ran-
somware, application-level DDoS
and cross-site scripting attacks,
compromised containers that
download extra malware or hunt
for sensitive data or weaknesses
in internal systems, container
breakouts that allow unautho-
rized access across systems,

22
that it was vulnerability CVE-
2017-5638, discovered months
earlier, that provided the way in.
The vulnerable code of CVE-
2017-9805 resides in the REST
plugin of the Struts framework.
The plugin fails to validate and
safely deserialize the user-up-
loaded data in the HTTP request.
This allows attackers to write
arbitrary binary code to the web
server and execute it remotely.
CVE-2017-5638 was reported
in the Jakarta multi-part pars-
er in March 2017. By injecting
a crafted Content-Type HTTP
header with a ‘#cmd=’ string, the
attacker is also able to execute
arbitrary commands on the web
server.
Ransomware
Recent ransomware attacks have
exploited various vulnerabilities,
including open ports, remote
code execution, and other appli-
cation liabilities.
WannaCry/WannaCrypt attacks
Microsoft Windows vulnerabili-
ties on SMB, RDP, and IIS service
ports like 445. Like most attacks,
there is a kill chain of events
that leads to the ransomware
demand.
Once attackers are able to get
inside a network, they spread
laterally, called an “east/west”
direction. Doing that has become
much easier in a containerized
environment due to the lack of
visibility into east/west container
traffic.
SambaCry is a seven-year-old
critical code-execution vulnera-
bility (CVE-2017-7494) discov-
ered in the Samba networking
software that allows a remote
attacker to take control of an
affected Linux system. If you
have your CVE database updated,
vulnerability-scanning software
could help to find this CVE in
container images. Other tech-
niques can also help detect a
privilege escalation or breakout.
Elasticsearch and MongoDB are
two of the most popular appli-
cation container downloads at
Docker Hub. Surprisingly simple
attacks involving open ports and
passwords could allow these to
be used for ransomware. Basic
container security precautions
could prevent such an attack.
Frequent auditing and testing
of host and container security
settings would highlight risks in-
troduced by the latest application
updates or host deployments.
Dirty Cow
The Dirty Cow (CVE-2016-
5195) kernel exploit is used by
attackers to overwrite a setu-
id program in the system. By
replacing the setuid program,
the attacker gains root privileges
when the program is executed.
While an attacker can perform
destructive acts, it is more likely
that they will try to expand their
access to other systems or appli-
cations as stealthily as possible.
It’s even possible for the exploit
to stick around in a container en-
vironment, even when the infect-
ed containers are destroyed.
Linux Stack Clash
The InfoQ eMag / Issue #70 / April 2019
The Linux Stack Clash vulner-
ability (CVE-2010-2240) was
recently discovered by Qualys
security researchers. They
called it “Stack Clash” because
it involves clashing the stack
with another memory region,
such as the heap. While this is a
local exploit in the user space, it
can still give attackers a foot-
hold into the operating system
or kernel. Stack Clash could be
used with other exploits to wreak
even more havoc on systems and
potentially move laterally. For
example, a recent sudo vulnera-
bility (CVE-2017-100367) could
be combined with Stack Clash to
be able to run arbitrary code with
root privileges.
Attack detection
Every successful attack involves
a kill chain of events, and de-
tection is possible at multiple
points using various techniques.
Generally, to mitigate these risks
in a container environment, the
following security measures
should be taken:
1. Scan hosts and containers
(before and during run-time)
for vulnerabilities and patch
vulnerable systems.
2. Monitor hosts and contain-
ers for suspicious processes
such as root-privilege escala-
tions and port scanning.
3. Detect lateral movements and
breakouts using suspicious
network connections to other
hosts or containers.
23

The InfoQ eMag / Issue #70 / April 2019
4. Lock down containers to
exclude external access if not
strictly required.
5. Continuously audit security
settings for containers, hosts,
and platform tools.
Curtailing these threats requires
a layered security strategy that
24
is implemented to prevent issues
throughout the stack and at
each phase of the deployment
process. The correct security
controls and automated tests to
detect and address attacks and
other issues are critical compo-
nents for ensuring effective and
secure applications.
Here are the measures you can
put in place throughout the
container deployment process to
enact continuous security.
Step 1: Build phase
Continuous security begins with
building secure applications.
Developers must understand and
use proper techniques for mini-
mizing risk within the application
code itself. These techniques
include:
• Code analysis — Developers
should use code-analysis
tools that can recognize
and report any potential
vulnerabilities.
• Hardening — Reduce the
attack surface of the con-
tainer image by removing any
libraries and packages that
aren’t needed and by restrict-
ing non-required functions.
• Image scanning — Just be-
fore the ship phase, container
images should be scanned
for vulnerabilities in order to
catch any issues before the
application is built. Registry
scanning should also be done
regularly, as this can find new
issues in existing images.
As a practical matter of execut-
ing these security measures,
they should be automated into
the delivery process/pipeline
using adequate tools, as doing so
ensures that they occur continu-
ously and without fail.
Step 2: Ship phase
When preparing to deploy im-
ages, security depends on the
right controls and verifications
in place to ensure trust. These
include:
• Image signing — This ensures
that the author/publisher is
verified and that no image
tampering has occurred. For
example, Docker’s Content
Trust feature provides this
security.
• User access controls — Sen-
sitive systems and de-
ployment tools (registries,
orchestration tools, etc.)
should have mechanisms in
place to effectively enforce
restrictions and monitor user
access.
In order to automate these secu-
rity features to facilitate continu-
ous security, images for produc-
tion use should automatically be
signed and tagged. Access con-
trols for sensitive tools should
be integrated with Active Direc-
tory or LDAP directories. Docker
Bench for Security can be run
as an automated process to test
content trust and access-control
issues.
Step 3: Run-phase preparation
The InfoQ eMag / Issue #70 / April 2019
Ahead of running application containers live in
production, the run-time environment needs to be
secured, including the host, kernel, Docker daemon,
and all network and system services:
• Host and kernel security — These can be safe-
guarded using security modules such as AppAr-
mor, SELinux, seccomp, or the equivalent host
security settings.
• System and Docker daemon security — Make
sure that only authorized users can access
sensitive systems and Docker commands and
settings.
• Secrets management — Use encryption to pro-
tect secrets that are subject to application and
API access.
• Auditing — Audit security settings both before
and during run-time. This can be accomplished
using the previously mentioned Docker Bench
for Security, Kubernetes CIS Benchmark, or oth-
er tools, and has the additional benefit of help-
ing to meet container compliance requirements.
• Orchestration and network security — Secure
the container environment by using container
management and orchestration tools capable
of controlling access, setting network policies,
and using appropriate names and labels. These
tools can also make sure that security contain-
ers are running on every host.
These orchestration and integrated security tools
can deliver automated security testing and checks
against standard benchmarks, as well as automatic
policy updates to adapt to configuration changes
and clearer visualization of behavior within the
application and containers.
25
 Step 4: Run-phase production Conclusion
The InfoQ eMag / Issue #70 / April 2019

Run-time in production environments is the most crucial phase when With the right strategy in place to
it comes to container security. This is the battleground where hackers integrate security tools and auto-
will attempt to exploit any vulnerabilities they’ve discovered, and se- mate policy, continuous contain-
curity efforts must detect and thwart those attempts. Hacker actions er security that offers protection
will include a kill chain, the series of events that occur as the hack- throughout the container deploy-
er moves from the host or another container to the targeted exploit ment process can be achieved.
point. This chain of events often presents several chances to detect
and prevent the hacker’s activities with these techniques:

• Network inspection and visualization — Inspecting network

behavior is the most opportune vantage point for monitoring
and recognizing suspicious activity. Be sure to inspect all con-
tainer-to-container connections as well as container and host
processes, and visualize application-stack behavior. If an attack
spreads and real-time response is necessary, these capabilities
will enable the security team to see it coming and respond.

• Layer-7-based application isolation — Go beyond layer-3 and lay-

er-4 network segmentation by also isolating and protecting based
on application protocols and inspecting container metadata. As
necessary, application activity can be blocked without affecting
containers.

• Threat detection — Monitor applications, always remaining on the

lookout for DDoS, DNS, or other varieties of network-based appli-
cation attacks.

• Privilege-escalation detection — Hacking efforts may attempt to

grant greater administrative privileges to unauthorized intrud-
ers. Detect these escalations on hosts and containers to foresee
breakouts and defend against such attacks. 

• Container quarantine — When a container exhibits suspicious

behavior, have the ability to quarantine it for safety while investi-
gating it.

• Run-time vulnerability scanning — All running containers should

be scanned for vulnerabilities; all newly created containers should
be scanned immediately as well.

• Packet capture and event logging — Capturing packets and creat-

ing event logs allows for more capable forensics efforts.

Tools for these security measures should be automatically deployed

to each host, and new container behavior should automatically be
learned or programmatically added to the security policy. Real-time
monitoring and alerts can be managed from either local consoles or
logs exported to SIEM systems.

26
 The InfoQ eMag / Issue #70 / April 2019
Developing a Secure
and Scalable Web
Ecosystem at LinkedIn
by Mira Thambireddy, information security engineer at LinkedIn
and James Baker, Senior Software Engineer at LinkedIn

LinkedIn’s hyper- Between 2008 and 2014, LinkedIn’s global member base grew from
growth placed strains 16 million members to 330 million members, straining our organi-
zation’s infrastructure. As our member base grew, we needed more
on the organization’s
insight into how our members were using our platform and what
infrastructure. A product lines we could introduce to best serve them.
new release model
was instrumental Scaling through hyper-growth
to scale and led to The company produced numerous internal, web-based applications
increased code quality, that engineers, product managers, executives, and operations teams
security, and member used to perform a variety of crucial tasks such as A/B testing, appli-
satisfaction. cation deployment and lifecycle management, reporting, analytics,
and more. Teams also undertook new approaches to solve technolo-
gy problems, introducing and vetting a number of different languages,
frameworks, and tools. Such growth and experimentation resulted
in a lack of uniformity among technologies and solutions between

27
 groups, which created strain as
The InfoQ eMag / Issue #70 / April 2019
engineers were hired to fill roles
within emerging teams.
Languages such as Python, Ruby,
Java, Scala, JavaScript, and
others emerged in various or-
ganic efforts or acquisitions that
peppered the ecosystem with in-
compatible but useful solutions.
Mind you, this exploration was a
healthy and intentional endeavor
to find the best long-term solu-
tions for problems we sought to
solve. Teams were encouraged to
seek out technologies that they
felt might benefit the organiza-
tion and this exploratory process
was instrumental in helping us
define what we would lean on,
long term, as a scaling and nota-
ble organization in Silicon Valley
and across the globe.
By mid-2015, several dozen
active projects had surfaced with
various implementations, frame-
works, and libraries. Because of
the varying approaches, teams
struggled to share code and,
though repositories and arti-
facts were in place, the imple-
mentations themselves lacked
uniformity. In the case of JavaS-
cript, some used a composite of
libraries and micro-libraries like
jQuery and Backbone.js; some
teams used popular frameworks
while others rolled their own. A
growing uncertainty loomed over
how we would approach building
front-ends for applications, how
developers could share common
logic across teams, and how we
could streamline best practices
28
for developer ergonomics and
satisfaction.
This growing number of varying
technologies also introduced a
security debt over time. Having
a large number of frameworks,
languages, and components
made it increasingly difficult to
assess the security posture of
the applications built on top of it.
Additionally, this undermined the
efforts to move towards releas-
ing a common framework-level
security solution to eliminate
certain classes of vulnerabilities.
The impact of a slow deployment
process
Around this same time, our
complex infrastructure for the
Linkedin.com website alone sur-
passed 3,000 submodules across
more than 6 million lines of code,
all within a single repository
trunk. This trunk approach was
governed by a tedious monthly
release cycle that encompassed
hundreds of engineers across
various teams. Prior to each
release, a release candidate (RC)
was selected and handed off to
our testing teams for four days of
manual regression testing. If they
found any bugs, we made hot-
fixes to the RC to avoid disrupt-
ing the deployment. Engineers
rushed to get their code checked
in before the deadline to avoid
having to wait an entire month to
deliver their features or bug fixes
to members.
Such a serial and time-sensitive
process had to be meticulously
timed with product manag-
ers and marketing partners to
coincide with their plans for new
feature releases. It wasn’t easy
to iterate on member feedback
within this infrequent cadence of
12 releases per year.
Further, prevention and remedi-
ation of potential security vul-
nerabilities strongly depended
upon the deployment and release
process. It was imperative that
once a fix was identified and in
place that we get it into the pro-
duction environment as quickly
as possible. This often meant
that security fixes were hotfixed
against the release cycle instead
of incorporated within it. It is
generally good practice to deploy
a security patch in isolation —
i.e., not to add other, non-security
bug fixes along with a securi-
ty update. This is primarily to
reduce the chances of re-intro-
ducing a security vulnerability if
the patch is rolled back due to a
non-security update that breaks
functionality.
A less obvious side effect of
hyper-growth, a long release ca-
dence, and a mixture of technol-
ogies was an emerging “blocky”
and inconsistent user experience
(UX). As LinkedIn began to em-
ploy user research in the product
development process, we found
that many users felt the site was
disconnected and that one page
would look different from anoth-
er. Because teams were releasing
so far apart in cycles, the feed-
back loop for UI changes was

Figure 1 / Our former deployment process.
delayed, impacting the quality of
changes over time.
Introduction of the 3x3 method
In 2014, LinkedIn’s mobile
engineering teams had begun
experimenting with what would
become our current release
model. Dubbed “train release”,
this approach shifted our month-
ly release cadence to one called
“3x3”, meaning that we would re-
lease three times per day, with no
more than three hours between
when code is committed to when
that code becomes available to
members.
The idea behind this was not
just web-specific. The ultimate
goal was to have all platforms
running on this cadence, includ-
ing iOS, Android, API, and other
back-end services necessary to
run the LinkedIn.com experience
The InfoQ eMag / Issue #70 / April 2019
and extended product lines and
services.
Transitioning to such a release
model was challenging and
required involvement from all
areas of engineering, especially
from our tooling and infrastruc-
ture teams. It meant that we
would have to revisit our existing
internal applications and tooling
to ensure that developers re-
ceived timely information about
the status of the deployment
process, that they had the proper
scripting and systems in place to
automate much of that process,
and that proper end-to-end test-
ing could take place to ensure
adequate code coverage.
We would not be able to test all
changes in such a short amount
of time between release cycles
in this newly proposed approach,
29
 placing a strong and necessary emphasis on test-
The InfoQ eMag / Issue #70 / April 2019

ing and automation. This, among other challenges,

resulted in the need for client-side technologies
that naturally emphasized testing in the develop-
ment lifecycle — testing historically has not been in
the purview of client-side engineering in the in-
dustry, particularly in web. This, combined with the
many pain points listed above, became the catalyst
for change in not only the Linkedin.com experience,
but in the structure of our infrastructure and appli-
cation-level technology stack.

While this meant that we could now fix a security

issue identified on the platform in a short time, it
also meant that security issues could slip in fast
with this frequency of code deployment. And when
such a model is adopted by 100+ applications that
comprise the LinkedIn ecosystem, it amplifies the
need for security automation.

Our next play

A simple but powerful approach towards secur-
ing web applications is to rely upon frameworks
that offer inherent security features. Of course, the
choice of framework is not only dependent on se-
curity, but also performance, ease of use, and other
factors.
Our infrastructure team, in collaboration with other
partner teams, began extensively researching and
vetting many languages, frameworks, and librar-
ies, creating gap analyses for technologies that
spanned the server-client relationship and the
applications that would serve as tooling for future
spin-ups in new and existing product lines and
internal platforms.
Alongside this effort, our User Experience Research
team established extensive focus groups and
feedback efforts from members to gain a better
sense of what the ideal experience for LinkedIn.
com would be.
These joint efforts of product, design, and engi-
neering resulted in Project Voyager - a mobile-first,
Figure 2 / Project Voyager.
consistent user experience (UX) across mobile
platforms (iOS, Android, and Web). Focusing on
mobile first gave us the opportunity to later extend
to a future desktop application that would embrace
the same UI patterns and theme, providing a con-
sistent experience across all platforms.
As a result of this effort, we chose two frameworks
for building our API and web client: the Play Frame-
work for Java and the Ember.js framework for web
were chosen to be the de facto frameworks used
for building web applications. LinkedIn had been
investing effort to contribute to the Play Framework
for some time prior to this new project. Our security
team performed an extensive gap analysis on the
security features that were currently available in
these frameworks against those features that were
required for our stack.

30

Figure 3 / Pre-commit hook tool for scanning for potential XSS.
Our analysis concluded that the
Play Framework provided a se-
cure-by-default approach along
with a responsive security team,
an active core-developer com-
munity, extensive documentation,
and a stable release cycle.
Ember.js shared all of these traits
as well. As a single-page appli-
cation (SPA) framework, Ember.js
also provided:
• future support for server-side
rendering (SSR) through tech-
nologies like Fastboot;
• developer-focused, CLI-
based ergonomics like gen-
erators and blueprints and an
active add-on community;
• an active core-developer
community;
• focus on impacting and
polyfilling emerging industry
standards through clear and
powerful abstract primitives;
and
The InfoQ eMag / Issue #70 / April 2019
• a well-developed and seman-
tic approach for testing UI
components.
By moving the web to a cli-
ent-side application, we were
able to establish a uniform
internal API for all clients (iOS,
Android, web), better aligning
our infrastructure across vari-
ous platforms and reducing the
number of applications needed to
provide data.
Ember.js’s focus on tests took us
another step towards automat-
ing deployment and was instru-
mental in our efforts towards
embracing 3x3. The framework
provides three types of tests:
integration, acceptance, and
unit tests. Integration testing
allowed us to test data flows
and interactions of components
in the application, acceptance
tests gave us user interaction
testing, and unit tests provided
us with ways to test application
logic. As developers generated
new components, the framework
would produce the test files as
31
 well, allowing the developer to
The InfoQ eMag / Issue #70 / April 2019
focus more on writing code and
verifying its functionality in the
tests as opposed to just in the
browser.
Security in depth
At Linkedin, our security team
performs in-depth design reviews
including hands-on penetra-
tion tests for all member-facing
products, features, and func-
tionalities. We are also heavily
invested in security automation.
However, with the 3x3 deploy-
ment architecture, we couldn’t
possibly scale manual penetra-
tion test for all builds, resulting in
a decision to double down on se-
curity automation. Once we find
a class of vulnerability that can
be detected with a high level of
confidence, we build automation
checks for that class. Our part-
ner product-security engineering
teams have helped in building,
maintaining, and scaling such
automation. This allows us to
Figure 4 / Our current (3x3) release cycle.
32
focus on more interesting areas
of the application and underlying
framework and provides us more
time to research some in-depth
vulnerabilities in those areas.
As we added API endpoints to the
application, we needed a security
analysis to prevent vulnerabili-
ties from emerging. Previously,
this process was operationally
cumbersome, given the number
of routes (paths to resources or
URLs) per application and the
number of applications in the
system. Our security team built
tooling to detect and notify us
of any new changes made to
an application, broken down by
the nature of change (addition
or deletion of an external API
route, modification to key code
of the application, etc.) to assist
in the evaluation of each such
commit to the application. This
let us determine the state of an
application since the last review.
This allowed for targeted reviews,
ensuring broader coverage and
faster turnaround time for the
assessment of our applications.
An established approach to
security assessment is through
the adoption of security in depth.
We do not want to be in a sit-
uation where the failure of one
particular security control results
in the failure of the entire chain.
We love to give developers the
tooling they need to avoid intro-
ducing security vulnerabilities.
Our product-security team built
tooling to scan the code changes
for potential vulnerabilities. If the
tooling detects any anti-patterns
or discouraged practices, it noti-
fies the developer and suggests
code fixes to allow the developer
the opportunity to properly ad-
dress the problem. Once past the
code review process, the tool-
ing again analyzes the code for
changes and if it finds a potential
vulnerability, it rejects the code
at the pre-commit stage of the
deployment pipeline, through the
adoption of custom pre-commit
hooks.
Once a security issue has been
identified on the platform through
any channel, our goal is to pre-
vent the same issue from resur-
facing in the future. To help avoid
regression issues, we build tool-
ing and test cases that contin-
uously run against the deployed
services, detect the reintroduc-
tion of a particular instance of a
security vulnerability, and alert
the security team.
In January 2014, prior to the
introduction of this system, we
identified over 5,000 potential
XSS vulnerabilities through code
scans; by January 2016, that
number was less than 500. The
number of pre-commit failures
due to the introduction of offend-
ing commits steadily declined
during this same timeframe.
Through our methodical ap-
proach to security automation,
we saw close to a 90% reduction
in the presence and introduction
of potential vulnerabilities.
To date, we average 50-75
commits in our web client per
day from a developer body of
over 100 UI engineers in a single
repository. Each of these com-
mits goes through a code review
system and requires several sep-
arate approvals from developers
on access-control lists (ACLs)
to ensure code is at the highest
quality. Code is evaluated against
a style guide of best practices
but is also scanned with linters an organization and has ushered
The InfoQ eMag / Issue #70 / April 2019
for various languages to ensure in increased code quality, secu-
that developers are writing code rity, productivity, and member
similar to their peers. As men- satisfaction. We are now more
tioned, these code changes also capable of providing our mem-
undergo an automated, custom- bers with a safer, faster, modern
ized security scan to check for experience, quickly resolving
known classes of vulnerabilities issues or bugs that are found and
including, but not limited to XSS, innovating more rapidly.
CSRF, and access-control issues.
Once developers receive the
approvals they need and have
addressed any open issues, their
code is submitted through a set
of systems to ensure its health
before it makes it into the set
of commits bound for the next
deployment.
These systems will perform a
variety of tasks as well as run all
of the application’s tests. If these
tests do not pass, they reject the
commit and notify the developer.
Once the commit passes, it en-
ters the trunk of the application
and a separate set of tests run to
ensure the code did not intro-
duce performance regressions or
exceed other obvious thresholds
established by the application
owners. Assuming it passes all
these, the commit ends up in a
production environment at the
next available deployment. If no
commits are introduced between
deployment times, a deployment
is still performed against the
same version; this is part of the
3x3 methodology and ensures
that code is rigorously tested.
The resulting shift towards a new
release model has been instru-
mental in our ability to scale as
33
InfoQ @ InfoQ
Curious about
previous issues?
This eMag explores how
Kubernetes is moving from
a simple orchestration
framework to a fundamental
cloud-native API and
paradigm that has
implications in multiple
dimensions, from operations
to software architecture.
InfoQ InfoQ
In this eMag covering .NET
Core, we will explore the
benefits of .NET Core and
how it can benefit not only
traditional .NET developers
but all technologists that
need to bring robust,
performant and economical
solutions to market.
This eMag will inspire you
to dig deeper into your
systems, question your
mental models, and use
chaos engineering to build
confidence in your system’s
behaviors under turbulent
conditions.