Professional Documents
Culture Documents
Configuration Management Best Practices: Who Am I?
Configuration Management Best Practices: Who Am I?
1
CM Best Practices Consulting © 2013
Who am I?
• CM Lead & Consultant for over 25 years
• Editor-in-Chief at CM Crossroads
• Author of CM Best Practices
• IEEE Management Board
• Tools and process agnostic
• The guy called in the middle of the night
when the release doesn’t work!
http://cmbestpractices.com
©
2013
2 April 9, 2013
1
9/9/13
2
9/9/13
Configuration
Management
• Configuration Identification
• Status Accounting
• Change Control
• Configuration Audit
3
9/9/13
Configuration
Identification
• Provides a specific and unique
identity to each configuration item (e.g.
binary, config file, documentation)
Status Accounting
4
9/9/13
Change
Control
• Establishing checkpoints including
gatekeeping (e.g. Production, QA, UAT)
and configuration control.
Configuration
Audit
• Inspect and identify the exact version
of any configuration item (physical &
functional)
5
9/9/13
Leads us to V & V
http://cmbestpractices.com
©
2013
12 April
9,
2013
6
9/9/13
7
9/9/13
CM
Functions
• Source Code Management
• Build Engineering
• Environment Configuration
• Change Control
• Release Engineering
• Deployment
8
9/9/13
Terminology
• Configuration items (CIs) include
binaries, source code, config files and
even documents
• ISO 1007 notes end user function
• Bob says, “anything where getting the
wrong version would be bad”
9
9/9/13
Principles
• Code is locked down and can never
be lost
• Code is baselined marking specific
milestones
• Managing variants using branches
• Code changed on a branch can be
merged
http://cmbestpractices.com
©
2013
19 April
9,
2013
More
Principles
• Processes are repeatable Agile and
Lean
• Traceability and tracking of all
changes
• Improves productivity and quality
10
9/9/13
Best
Practices
• How do we establish source code
management that adheres to these
principles?
• Better question is how does CM add
value and help facilitate the
development effort?
Sandboxes
• Provide a degree of isolation
• Support multiple sandboxes
• Allows the “what-if” scenario
• Cheap and disposible
• Make sure that you refresh before
commiting your code
11
9/9/13
CopyBranches
• Example of a copybranch (versus
delta)
12
9/9/13
Handling
a
bugfix
• We need to change Revision 2, but 3
is already being developed
Inner
Merge
• You need to merge the change on the
bugfix branch back to the main trunk
13
9/9/13
Outer
merge
• You also might want some new code
merged from trunk to the bugfix
Software
Patterns
• Fixing bugs while developing next
version of a product in parallel
• Support for developers working in
parallel
• Track component baselines
Software Configuration Management
Patterns By Steve Berczuk
14
9/9/13
Streams
• Provides a clear usage paradigm
• Model components and architecture
• Control flow of changesets
• Snapshots create baseline of code
• Ability to load a particular snapshot
• Strong security authorization and
entitlements
• Complete history and traceability
http://cmbestpractices.com
©
2013
29 April
9,
2013
Examples
• Organize code into components
• Use Streams & branches
• Make merging viable and traceable
• Navigate your repository metadata
• Use Tasks to track your work
15
9/9/13
16
9/9/13
Training
• Training is the “hill to die on”
• Best when given by your CM support
team
• Includes the process you want them
to use
• Much more than just vendor training
• Test first and then teach
http://cmbestpractices.com
©
2013
34 April
9,
2013
17
9/9/13
Future
• More robust Application Lifecycle
Management solutions
• Integration with the entire ALM
• Open standards (OSLC)
• Toolchains for everyone!
18
9/9/13
Build
Engineering
• Reliable and repeatable automated
process to compile, link and package
code components.
• Must handle complex compile
dependencies
• Continuous integration (or nightly
build)
• Visibility into who broke the build
http://cmbestpractices.com
©
2013
37 April
9,
2013
Principles
• Builds are understood and repeatable
• Builds are fast and reliable
• Every configuration item is identifiable
• Source and compile dependencies
can be easily determined
19
9/9/13
More
Principles
• Code should be built once, but
deployed anywhere
• Build anomolies are identified and
managed
• The cause of broken builds is quickly
and easily identified and fixed!
CI
Identity
Crisis
• Who am I?
• What if you cannot reach the version
control system (VCS)?
• CIs should be identifiable outside of
the VCS
• Breadcrumbs are not enough
• Its tagged so I can build it - right?
(not so fast – maybe you can't!)
http://cmbestpractices.com
©
2013
40 April
9,
2013
20
9/9/13
Version
IDs
• You need to embed an immutable and
unique version ID
• You must have a procedure to easily
pull out the version ID at runtime
• Cannot depend upon the version
control system (VCS)
• Stamp in the tag or label
http://cmbestpractices.com
©
2013
41 April
9,
2013
21
9/9/13
Compile
Dependencies
• Get environment variables understood
• You must be able to build at the
command line
• Developers forget what they set in the
IDE
• What was that classpath?
• What libraries did I use?
http://cmbestpractices.com
©
2013
43 April
9,
2013
Independent
Build
• Code must be built using a different
computer account on seperate
computer
• First time it always fails !
• You have to find whatever they forgot
to check into version control
• Verification and validation of the build
• Satisfy regulatory requirements (audit)
http://cmbestpractices.com
©
2013
44 April
9,
2013
22
9/9/13
Continuous
Integration
• Framework for structuring the entire
build, package and deploy
• Determine build and integration issue
early in the process
• Should include deployment to a test
environment
23
9/9/13
Code
Analysis
• Source code repository helps code
analysis in two different ways
• Static Code Analysis by providing a
repository
• Instrument the code using variants
• Security Audits
• Uncover code defects
• Code coverage
http://cmbestpractices.com
©
2013
47 April
9,
2013
Build
Frameworks
• Build agents
• Preflight builds
• Allows use of the build farm
• Moves the build framework upstream
• Supports rapid iterative development
24
9/9/13
25
9/9/13
DevOps
• Set of Principles where development
and operations partners
• Better communication
• Knowledge sharing
• Moving build and deploy upstream
Future
• Focus on complete deployment
framework
• Support rapid iterative development
• Virtualization and cloud computing
leads us to very fast build, package and
deploy
• Don't forget the automated testing
http://cmbestpractices.com
©
2013
52 April
9,
2013
26
9/9/13
Environment
Configuration
• Managing the environments including
creation and controlled configuration
• Procedures to manage compile and
runtime dependencies (e.g. database
access)
• Monitoring runtime environment for
unauthorized changes (e.g. ports open)
Principles
• Environment configuration
dependencies are identified and
understood
• Environments can be interogated for
current status
• Code is built once and configured
using automated procedures
http://cmbestpractices.com
©
2013
54 April
9,
2013
27
9/9/13
More
Principles
• Environment configurations should be
changed in a controlled and predictable
way
• Environment configurations should be
documented and understood by all
28
9/9/13
Using
Tokens
• Your code should read $DB1
• Substitute the correct database
• Centralize environment variable
assignment
Configuration
Control
• Should identify and control
environment configuration
• Trust but verify
• Good example of where a CMDB can
help keep surveillance during runtime
• Depends upon Change Control
29
9/9/13
Future
• Cloud computing and virtualization
are having a huge impact
• Full size environments created on the
fly
• Sharing a pool of resources
• Can I rent a super computer for a few
days please?
http://cmbestpractices.com
©
2013
59 April
9,
2013
Change
Control
• Management of changes including
gatekeeping and configuration control
• Process related changes managed
through the SEPG
• Change Advisory Board (CAB) to
evaluate the downstream impact of a
change
• A priori change control
http://cmbestpractices.com
©
2013
60 April
9,
2013
30
9/9/13
Principles
• Changes should be planned and not
just last minute events
• Changes should be understandable,
including their downstream impacts
• Authority and approvals for changes
should be established and obtained as
appropriate
http://cmbestpractices.com
©
2013
61 April
9,
2013
More
Principles
• Procedures for emergency changes
should be established to cover
emergency incidents
• Change control should assess and
confirm that all configuration
management processes are being
followed
http://cmbestpractices.com
©
2013
62 April
9,
2013
31
9/9/13
A
Priori
• May I have permission to make that
change?
• Facilitates an ALM task based
approach
• Who said you could work on that?
• Common for defense contractors
• My friends at the FAA
http://cmbestpractices.com
©
2013
64 April
9,
2013
32
9/9/13
33
9/9/13
e-‐Change
Control
• Routine changes
• Still requires traceability and
transparency
• Good way to implement the change
advisory board (CAB)
34
9/9/13
Retrospective
• After action review
• Need open and honest evaluation
• Opportunity to improve the process
• Drives the entire release process
35
9/9/13
Release
Management
• Consists of Release Engineering and
release coordination (PMO)
• Packaging and identification (e.g.
manifest) of all components built in the
build engineering function
• Automation to build, package, stage
and deploy releases
• Don't forget being able to rollback!
http://cmbestpractices.com
©
2013
71 April
9,
2013
Principles
• Releases should be readily
identifiable with an immutable version
ID
• Releases should be packaged with all
dependencies identified and controlled
• Packaging should be automated and
and designed to avoid human error
http://cmbestpractices.com
©
2013
72 April
9,
2013
36
9/9/13
More
Principles
• Release management should be fast
and reliable to facilitate iterative
development
• Must be able to audit the release
package
• Contents of release trackable (audit)
• Release management source of
information on status of all releases
http://cmbestpractices.com
©
2013
73 April
9,
2013
Manifest
• Documents contents of release
package
• Embedded (immutable) version ID
• Requires procedure to retrieve
version ID
• Created through automated
procedure
• Verifiable through configuration audit
http://cmbestpractices.com
©
2013
74 April
9,
2013
37
9/9/13
Release
Maps
• Complete list of release contents with
MAC-SHA1 or MD5 hash
• Utility to recreate release map and
compare to version shipped with
release
• Should be able to verify MAC-SHA1
or MD5 hash
http://cmbestpractices.com
©
2013
75 April
9,
2013
38
9/9/13
Release
Coordination
• More of a PMO function that works
closely with Change Control
• Release Calendar is essential
• Track requirements completed in
release notes
• Communicate status of release to all
stakeholders
http://cmbestpractices.com
©
2013
77 April
9,
2013
Staging
• Essential practice that ensures the
success of the deployment
• Should be fully automated
• Must be fully traceable
• Configuration audit verifies successful
completion
39
9/9/13
Future
• Complete release and deployment
framework
• Both open source and commercial
solutions
• Integration with the ALM
• Status of my deploy on the dashboard
Deployment
• Should be the smallest of the
functions
• Should be engineered to be a push
button lightswitch
• Requires full traceability
• Run by Ops
• Rollback is essential
• Relies upon release engineering
http://cmbestpractices.com
©
2013
80 April
9,
2013
40
9/9/13
Principles
• Promoting a release should be as
simple as possible
• Backing out a release should be as
important as promoting
• Promoting a release should be fully
traceable with an audit log of all
changes
http://cmbestpractices.com
©
2013
81 April
9,
2013
More
Principles
• Only Ops should be involved with
deployment
• Separation of controls essential for
compliance
• Unauthorized changes should be
detected
• Configuration audit to verify
• Retrospective & ongoing improvement
http://cmbestpractices.com
©
2013
82 April
9,
2013
41
9/9/13
I
Make
Mistakes
• But I will always be able to tell you
what mistakes I made
• Full traceability
• Few steps, if any, are done manually
• Verification that steps were completed
correctly is essential
• Automate everything!
http://cmbestpractices.com
©
2013
83 April
9,
2013
42
9/9/13
Smoke
Testing
• Last step of the deploy is always
testing
• CM should be part of the QA and
testing function
43
9/9/13
Changing
Landscape
• Cloud computing and virtualization
• OSGi – plug and play
• Application servers to handhelds
• Can we get that build & deploy done
now?
• Continuous Integration becomes
Continuous CM
http://cmbestpractices.com
©
2013
87 April
9,
2013
Paradigm
Shift
• In many organizations deployment means
giving up your weekend
• We need to shift the way that we look at
deployment
• Making the deploy a non-event
• You don't believe so I will quote a few
colleagues
http://cmbestpractices.com
©
2013
88 April
9,
2013
44
9/9/13
45
9/9/13
CM
for
Agile
CM that is adapted to suit the continuous
nature of change that Agile provides
without sacrificing the values of CM.
46
9/9/13
47
9/9/13
48
9/9/13
49
9/9/13
50
9/9/13
Puppet/Chef
• Automate provisioning, patching and
configuration of operating system and
application components
• Systems integration framework
• Scalable and extensible
• Used in other deployment frameworks
www.puppetlabs.com www.opscode.com
Continuous Deployment
• Rapid iterative deployment
• Automate everything
• Keep the deployments small
• Minimize risk
• Easier to deal with problems
• Ability to fall back is important
51
9/9/13
Common Problems
• Deployments can be risky
• Missing a single step can result in
problems
• Too many mistakes
• Takes too long
• No way back
• Assumed defeat
Deployment
Pipeline
A deployment pipeline is … an
automated implementation of your
application’s build, deploy, test and
release process
52
9/9/13
Antipatterns
• Deploying Software Manually
• Deploying to Production-like
environment only after Development is
complete
• Manual Configuration of Production
Environments
Continuous Deployment, p. 7 – 10
http://cmbestpractices.com
©
2013
106 April
9,
2013
53
9/9/13
54
9/9/13
Devops
Dev/QA
Focus
• Development
• QA & Testing
• Operations
• Self Managing/Organizing Teams
55
9/9/13
New
Term
• Group of concepts
• Been around for a while
• Use case is compelling
• Stimulating discussion
• Necessary to meet demand
56
9/9/13
Portmanteau
• Combination of two words
• Development
• Operations
57
9/9/13
58
9/9/13
Release Antipatterns...
http://cmbestpractices.com
©
2013
117 April
9,
2013
Release
Antipatterns
l Deploying software manually
l Deploying to a production-like
environment.
59
9/9/13
Dev/QA
Focus
• Development
• QA & Testing
• Operations
• Self Managing/Organizing Teams
60
9/9/13
61
9/9/13
CM/Devops
• Flexible technical background
• Good knowledge of development
• Knowledge of QA/Ops
• Strong automation skills
• Some systems administration
• Ability to work across silos
Toolsmith/Devops
• Strong technical background
• Strong scripting skills
• Diving deep into the tools including
troubleshooting
• Understands toolchains and finds
flexible solutions
• Process orientation – focus on
traceability
http://cmbestpractices.com
©
2013
124 April
9,
2013
62
9/9/13
Sox
Compliance
• Section 404 of the Sarbanes Oxley
Act of 2002
• Using ISACA Cobit 4.1
• 34 high level IT controls
• PCI compliance
• SAS-70
63
9/9/13
ISO 9001
• Establishes the quality management
system
• ISO 90003 is the software standard in
the 9000 family of standards
• Uses ISO 12207 (or 15288) to specify
lifecycle processes
• ISO 10007 for CM
• IEEE 828, EIA 649-A, Mil Std coming!
http://cmbestpractices.com
©
2013
127 April
9,
2013
Which
Standards?
• IEEE 828 – CM Planning
• EIA 649-B – Non compliance
• ISO 90003 to support QMS
• Full lifecycle ISO 12207
Tailor !
64
9/9/13
Moving
Upstream
• Dev to CM to QA to Ops
• Cross functional focus
• Speed up development
• Build a great deployment architecture
• Give it to Devs as a service!
Frameworks
• ITIL v3 including CMDBs, federated
CMDBs, CMS, DML…
• Cobit for SOX
• CMMI ->>>> Agile
65
9/9/13
Assessment
• First step is to assess current
practices - “As-Is”
• Compare to industry standards and
frameworks
• Determine “To-Be”
• Create a plan for improving your CM
processes
http://cmbestpractices.com
©
2013
132 April
9,
2013
66
9/9/13
Configuration
Management
• Configuration Identification
• Status Accounting
• Change Control
• Configuration Audit
67
9/9/13
68
9/9/13
Configuration
Management
Best
Practices
137
CM Best Practices Consulting © 2013
69