Application Security &

DevOps
How DevOps can keep
application security from
being an afterthought

A lot of Dev and Ops people look at their security teams with
disdain. "Those guys are just blockers... All they do is throw
obstacles in my way and say no. They'll never be DevOps or
even Agile." In return many security people see the emergence
of DevOps as creating a wilder, less managed environment in
which they would face more risk and greater security
challenges.
--- James Turnbull, Puppet Labs
I recognize that my code will be used in ways I cannot
anticipate, in ways it was not designed, and for longer
than it was ever intended.
--- Rugged Manifesto (Josh Corman, David Rice, Jeff Williams)
My Observation: prevailing view of security as separate
from development and operations, DevOps or no DevOps.

Application Security is a 2nd Class

Citizen
PROBLEM EXAMPLES:
SSL is a requirement, but may be addressed very
late in the SDLC. Coding for SSL and non-SSL may
differ substantially.
LDAP authentication (authc) and/or authorization
(authz) is a requirement, but ops may not find time
to supply a dev/test directory server with realistic
entries. Or any dev identity server at all
Automated tests on web pages test happy-path
function, maybe even a few data validation edge
cases, but often ignore security tests.

Application Security is a 2nd Class

Citizen contd
No firewalls in dev plenty of em in prod.
Arguments over ports at 11th hour.
Third-party libraries not subjected to security
audits.
Production system trust boundaries differ from
development
DBAs and development team have widely
diverging views on authentication.
Organizational security chief may have pet
theories about IT security best ask early.
End users want SSOnobody asked them.

History of DevOps
DevOps comes from a need for innovation on the systems side of
technology work.
ESM (Enterprise Systems Management)
mid-2000s
ITIL as a governance framework largely superseded by ITIL Lite Visible
Ops approach
shift from large vendor focused to more open source and smaller vendor
offerings.

first OReilly Velocity conference 2008,

focus on Web performance and operations
2009 seminal presentations about the developer/operations collaboration at
large. Provisioning tools like Puppet and Chef.

Parallel thinking about Agile Systems Administration

Focus on analogies from Kanban/Lean manufacturing processes -> IT
systems administration.
2009: popularization of the term DevOps.

What is DevOps?
More collaborative relationship between
development teams and operations teams
System administrators participating in an
agile development process alongside
developers
Fusion of:
Toolchains of monitoring and provisioning tools,
CI, automated testing etc
agile processes
Development teams /operations team cooperation

Secure SDLC

Security requirements need to be defined as early as possible during the

SDLC

Standard Phase

Security Task

Notes

Kickoff

Organizational security Management support

policy
for security

Requirements

Project security policy

Security model
Security Specification

Design

Defensive threat
modeling

Development

Security code reviews

Includes TDD and unit

test

Testing

Adversarial threat
modeling (pen-tests)

Functional, integration,
system, UAT

Implementation

Security configuration

Smoke tests

Ops & Maintenance

Security maintenance
checklist

Decommissioning

Back-out security
configuration

E.g. dedicated LDAP

groups

Some Secure SDLC

Examples
Some processes and standards that can support
security in a SDLC:
Microsofts Trustworthy Computing Security
Development Lifecycle
SEI Team Software Process for Secure Software
Development (TSP)
Correctness by Construction
Merging security assurance activities into Agile
development methods (recent)
The Common Criteria (ISO/IEC 15408)
Software Assurance Maturity Model

Many common processes and methodologies are

not designed specifically to address software
security from the ground up.

Problems with Secure SDLCs

One of the major hurdles is the availability of security expertise for

the developer.
2009 Veracode survey:
Only 13% of companies know the security quality of critical
applications.
60% are increasing scrutiny on partners for delivering secure
code
Only 34% of companies have a comprehensive SDLC process
which integrates application security.
57% of organizations dont have AppSec training programs for
their developers.
64% of respondents cannot afford desired application security.
SSE-CMM, Trusted CMM, Common Criteria, Correctness by
Construction, and TSP Secure require security-knowledgeable
resources and that an organization adopt a different and more
rigorous development approach.
Organizations are showing increased response to security, but
there is still a long way to go before considerations of security in

Typical Process Conflicts

Operations often not involved in architectural decisions or code

reviews.
E.g. development team comes up with some system interfaces that ops
hears about well after the fact. Whaddya mean youve got that
external party coming in that way?

Developers often cannot identify minimal steps to reach a

configuration state.
E.g. trial and error to set up working security, little idea of what is
actually needed.
Developers not given time to determine essential elements of a
vulnerability fix.

Developer configurations are optimized for development, not

production.
E.g. no LDAP, no SSL, inadequate development DBs.
Development typically has no configuration management, test might
have, prod usually does.
An entire developer system may be on one server, where prod uses
four.

Development and operations do not cooperate to define the

Secure DevOps Challenges

Traditional DevOps
developers want change, operations wants
stability
Both give security short shrift
DevOps - mixing of infrastructure operations
with software development, in short release
cycles squeezes out security, which
requires deliberation

Current DevOps is often a dyad

Secure DevOps is a triad

Goals for Secure DevOps

Agile Security Testing
Secure Coding + Secure Operations + Secure
Collaboration
Developer training now has meaning
Faster communication between Development and
Operations on Vulnerability Information
Faster patching/closure of vulnerabilities
Process of collaboration between Development
and Operation
Single manager/management system for security
during the release cycle

Goals for Secure DevOps (2)

Identify security responsibilities:
Operations is responsible for architecting the more stable
infrastructure components.
Agile developers handle the application, which is now a 1 st class
security concern along with infrastructure.

Developers work within a secured environment that is locked

down to production standards.
Establish a security standard for the (infrastructure) baseline
that is monitored from development to production.
End results:
accountability for securing the whole stack
significantly reduce the impact of security audits on the DevOps cycle
High degree of automation, hence efficiency

Engage your QA/QC people

Automating Security
What to automate
SAST (Static Application Security Testing)
Examine source code, byte code or application binaries
for conditions indicative of a security vulnerability
Involves tools that help with either a) Review or b)
Static Analysis

DAST (Dynamic Application Security Testing)

Black-box (Functional and non-functional), White-box,
and Defect-based tests.
Examine the application in its running state in order to
discover security vulnerabilities
Robustness testing (i.e. fuzz testing) or fault-injection

Automating Security (2)

Introduce security automation gradually
Low-hanging fruit
Add static code analysis to CI system, clearly
distinguish from non-security

Require that functional tests incorporate

authc and authz
E.g. QTP or Selenium web page tests must hook
into a security realm, even if its a mock.
Must test method authorizations in development

Engage your CI enthusiasts

Separation of Duties
Deconflicting DevOps and traditional
attitudes
Management support for secure
SDLC/secure DevOps/Rugged DevOps
process
Single boss for security work across Dev
and Ops, with management and
business backing
IT Separation of Duties is fine as far as it
goes, but dont let an overzealous admin
misinterpret it.

Conclusions
Identify and engage champions and process
enthusiasts
Operations owns static infrastructure baseline;
development owns application
Single manager overseeing security
Developers have same environment as production
Development, test and production all use security
Build security into automated testing and
provisioning
Train your people developers and operators in
security

Notes
DAST. Fault-injection can include Trusted
Computing Base (TCB) testing including the
loading of shared libraries or DLLs at run-time.
Dynamic testing can include concurrency
checking and many other factors.
Tools:
SAST: HP/Fortify, IBM, Veracode, Checkmarx,
Grammatech, Amorize, Coverity, Klocwork and
Parasoft, FindBugs, PMD and FxCop
DAST: WebScarab

Resources
http://architects.dzone.com/articles/short-history
devops
https://buildsecurityin.us-cert.gov/bsi/articles/k
nowledge/sdlc/326BSI.html
ITIL Lite, Malcolm Fry,
http://www.best-management-practice.com/Publicat
ions-Library/IT-Service-Management-ITIL/ITIL-200
7-Edition/ITIL-Lite-A-Road-Map-to-Implementing-P
artial-or-Full-ITIL
/
http://www.infoq.com/news/2010/06/rugged-soft
ware-