The Future of DevOps Toolchains Will Involve

Maximizing Flow in IT Value Streams

Published: 14 January 2020 ID: G00464224

Analyst(s): Manjunath Bhat, Daniel Betts, Hassan Ennaciri, Chris Saunderson, Thomas Murphy

DevOps toolchains are changing, and the discrete automation silos of the
past are evolving into platforms that orchestrate application delivery as a
value stream. Infrastructure and operations leaders should revisit their
toolchain strategies to meet digital business demands.

Key Findings
■ Gartner client inquiries reveal increased interest in platform-centric approaches to building
DevOps toolchains that support end-to-end capabilities for continuous delivery.
■ Product teams see a greater need for unified visibility, orchestration, integration, governance
and management of the DevOps value stream to improve flow and traceability.
■ The changing nature of applications, due to cloud adoption, machine learning, commercial off-
the-shelf and open-source software, requires specialized and best-of-breed DevOps toolchain
capabilities.

Recommendations
Infrastructure and operations leaders responsible for selecting and deploying DevOps toolchains
should:

■ Drive business agility by using DevOps value stream delivery platforms that reduce the
overhead of managing complex toolchains.
■ Maximize flow in DevOps value streams by using DevOps value stream management platforms
for integration, orchestration, automated compliance and value stream mapping.
■ Support rapid innovation by investing in best-of-breed tools to fill niche gaps in software
delivery involving machine learning models and container orchestration.

Table of Contents

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Strategic Planning Assumption............................................................................................................... 2
Analysis.................................................................................................................................................. 3
How Value Streams Relate to DevOps.............................................................................................. 4
Value Stream Mapping......................................................................................................................5
DevOps Value Stream Delivery Platforms.......................................................................................... 6
DevOps VSDPs Are Extensible by Design................................................................................... 7
Benefits of a Platform Approach................................................................................................. 7
DevOps Value Stream Management Platforms..................................................................................8
Difference Between DevOps VSMPs and Enterprise Agile Planning Tools....................................9
Differences Between DevOps VSMPs and DevOps VSDPs.........................................................9
Critical Capabilities of a VSMP.................................................................................................. 10
Best-of-Breed DevOps Tools Support Niche and Emerging Use Cases.......................................... 12
Cloud-Native Tools................................................................................................................... 12
Security/Compliance Tools to Protect Against New Threat Vectors........................................... 12
Tools to Operationalize MLOps................................................................................................. 12
Open-Source Software Management Tools...............................................................................13
COTS Application Management Tools.......................................................................................13
Conclusion..................................................................................................................................... 13
Gartner Recommended Reading.......................................................................................................... 13

List of Figures

Figure 1. Three Ways Forward for DevOps Toolchains............................................................................ 4

Figure 2. DevOps Optimizes the Flow of Value........................................................................................5
Figure 3. DevOps VSDPs........................................................................................................................6
Figure 4. Platforms Allow Teams to Focus on Customer Value................................................................ 8
Figure 5. DevOps VSMPs..................................................................................................................... 10

Strategic Planning Assumption

By 2023, 70% of organizations will use value stream management to improve flow in the DevOps
pipeline, leading to faster delivery of customer value.

Page 2 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Analysis
DevOps relies on tools and technologies that automate repetitive aspects of software delivery for
speed and consistency. In 2018, Gartner defined the market for application release orchestration
(ARO) tools, which enable organizations to scale release activities by orchestrating deployments,
managing pipelines and environments, and providing governance to improve quality and velocity.

Four overarching trends are disrupting the ARO market:

■ There is increased interest in a unified platform to simplify the complexity of integrating pipeline
activities across the DevOps value stream. The DevOps value stream comprises a set/sequence
of activities product teams undertake to design, develop and deliver a service in response to a
customer need, using DevOps.
■ Organizations are facing challenges in assessing and improving flow metrics in application
delivery — for example, release velocity, product team efficiency, lead time and cycle time.
■ The adoption of container-based architectures for deployment and orchestration tears down the
wall between continuous integration (CI) and continuous delivery (CD) tools. The emergence of
the CD Foundation (as part of the Linux Foundation) creates a vendor-neutral home to support
CI/CD for cloud-native applications.
■ The use of a unified platform encourages end-to-end automation (“systems thinking”), as
opposed to automating in disparate silos (“local optimization”). There is the downside of
suboptimal capabilities for niche use cases, such as machine learning (ML) and serverless
architecture, and that’s where best-of-breed tools play a role.

In response to these trends, DevOps toolchain providers have started taking platform-centric
approaches. They are aimed at expanding into adjacent capabilities on the upstream (e.g., agile
planning, version control and integration pipelines) and downstream (e.g., support for
programmable infrastructure and automated policy governance for better change management and
audit compliance).

ARO no longer provides the correct perspective to inform buying decisions in a rapidly evolving and
expanded vendor landscape. Therefore, Gartner will retire the Magic Quadrant for ARO. During the
past year, ARO providers have expanded their offerings or consolidated through mergers and
acquisitions. Gartner client conversations indicate increased interest in tools that encompass both
development and operations capabilities.

This research provides insight to help application leaders and infrastructure and operations (I&O)
leaders navigate these trends, and highlights the three ways forward (see Figure 1).

Gartner, Inc. | G00464224 Page 3 of 16

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Figure 1. Three Ways Forward for DevOps Toolchains

The three ways forward are not mutually exclusive — they can and will coexist in most
large organizations.

How Value Streams Relate to DevOps

DevOps emphasizes people, culture and collaboration between development, operations and other
stakeholders to improve the delivery of customer value. DevOps value stream is the sequence of
activities required to deliver customer value through software using agile and DevOps practices.

“If you can’t describe what you are doing as a value stream, you don’t know what
you’re doing.”

— Karen Martin and Mike Osterling, Value Stream Mapping

DevOps practices improve the flow in the value stream through agile delivery methods,
collaboration and automation. Continuous feedback is an important aspect of value streams,
because it helps remove constraints and, thus, improves quality, reliability and safety. Product
teams obtain feedback through various means — CI provides quick feedback on code quality,
stability and potential to ship; CD provides feedback on product quality, usage, adoption and
reliability in production (see Figure 2).

Page 4 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Figure 2. DevOps Optimizes the Flow of Value

Without a value-stream-based approach, product teams will resort to local optimizations to improve
their individual efficiency and work quality. However, these local optimizations do not result in overall
system improvement, because bottlenecks continue to persist in other areas of the value stream.
Worse, it can introduce problems in other phases. For example, product teams that follow agile
practices without implementing DevOps fall short of the full potential for improving release cadence.

Improvements in faster development cycles create a huge deployment queue for operations staff
unprepared to handle the increased flow. Longer deployment times stifle continuous product
improvement, because development teams do not receive quick feedback from production
environments.

Value Stream Mapping

DevOps value stream delivery and management platforms provide unique visibility and traceability
into workflows throughout the DevOps value stream, because they plug into planning, development
and operational tools. The platforms provide visualization tools to analyze customer-focused
metrics, such as lead time, deployment frequency, defect escape rate, feature adoption and time to
respond to failures. The visualization of how work flows through the plan, build, test, preproduction,
release, configure and operate phases constitutes a value stream map.

A value stream mapping exercise builds trust, creates transparency and aligns team’s goals with
organizational objectives. It encourages systems thinking by depicting performance at a team level
and discourages a narrow focus on improving operational metrics in a specific silo.

Gartner survey data supports the need for value stream mapping. The Gartner 2019 DevOps Survey
findings reveal that 72% of respondents use between five and 35 toolchains for their DevOps
efforts. Seven percent of respondents report between 51 and 100 toolchains.1 Gartner analysis of

Gartner, Inc. | G00464224 Page 5 of 16

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 social media conversations2 indicates that conversations on DevOps toolchains are shifting from the
“adoption of tools for CI/CD pipelines” to “innovations for seamless integration throughout the value
stream.” These conversations highlighted the need to maximize flow in the DevOps value stream,
minimize bottlenecks, and enhance team productivity, employee morale and customer satisfaction.

DevOps Value Stream Delivery Platforms

DevOps value stream delivery platforms (VSDPs) enable buyers to leverage the end-to-end
capabilities of a DevOps toolchain from a single provider. VSDPs aim to deliver the capabilities
required to take an idea from concept to production, and back to ideation, through continuous
improvement and feedback. VSDPs look to address integration challenges in building the DevOps
toolchain; however, they may not offer best-of-breed tools across the entire pipeline (see Figure 3).

Figure 3. DevOps VSDPs

Through native support or APIs for extensibility, VSDPs provide a subset of the following
capabilities:

1. Plan and Create

■ Plan releases and sprints
■ Source code and artifact repository
■ Code editors and peer review tools; build automation capabilities
■ Manage product roadmap and backlog as part of iterative planning
■ Feature flag management support

Page 6 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 2. Integrate and Verify
■ Software composition analysis for open-source security and license compliance
■ Adherence to secure coding guidelines and regulatory standards through continuous
compliance automation
■ Software test automation (functional and nonfunctional)
■ Prebuilt connectors, custom APIs and webhooks by being extensible
■ Support for CI
3. Deploy and Operate
■ ARO — release orchestration and deployment of applications across all environments —
development, test, staging and production
■ Infrastructure automation (configuration management and immutable infrastructure support)
■ Automated change approval policy
■ Mitigate deployment risks using predictive analytics, feature flags and canary deployments
4. Monitor and Improve
■ By harnessing data from full-stack monitoring tools
■ By managing issues, bugs and incidents
■ The ability to do root cause analysis, event correlation and anomaly detection
■ DevOps flow metrics, such as flow velocity and flow efficiency, and reliability metrics, such
as change failure rate and time to restore service

DevOps VSDPs Are Extensible by Design

Beyond the native capabilities built into the platform, VSDPs are extensible by design and integrate
with third-party tools and extensions to:

■ Integrate with specialized tools to either augment or complement functionality

■ Expose toolchain data to other tools — for example, DevOps value stream management
platforms (VSMPs)
■ Orchestrate workflows

Benefits of a Platform Approach

A platform approach to toolchains should result in a significant, rather than an incremental flow of
value. Gartner’s definition of DevOps VSDPs does not describe a system that cobbles together
disparate DevOps tools. Instead, we view VSDPs as integrated platforms that enhance productivity,

Gartner, Inc. | G00464224 Page 7 of 16

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 collaboration, communication and business outcomes for the product team through the interactions
among the platform components.

“A system is not the sum of the behavior of its parts; it’s the product of their
interactions.”

— Russell Ackoff

DevOps VSDPs and VSMPs (as we will see below) provide intangible benefits to the product teams.
They allow product teams to focus on customer value-adding activities, instead of operational
tasks, such as building and maintaining the toolchain. The access to unified analytical insights
across the toolchain enables product owners and business stakeholders to assess the level of
business risk, release cadence, responsiveness to change and cross-team collaboration (see Figure
4).

Figure 4. Platforms Allow Teams to Focus on Customer Value

DevOps Value Stream Management Platforms

DevOps VSMPs enable organizations to integrate and orchestrate the end-to-end value stream
capabilities from multiple providers.

Product teams face challenges to quantify and improve release velocity, application quality,
customer value and organizational effectiveness. This creates the need for an orchestration control
plane (management and governance layer) to plan, build, release, monitor activities, and have end-
to-end visibility and analytics across the complete value stream.

Page 8 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Difference Between DevOps VSMPs and Enterprise Agile Planning Tools
VSMPs have some overlap with enterprise agile planning (EAP) tools. Although a few EAP tools do
provide visibility into DevOps pipeline and support value stream mapping, their core capabilities and
focus include backlog management, Scrum and Kanban team support, product planning, Scaled
Agile Framework (SAFe) support and Epic-based forecasting. On the other hand, VSMPs focus on
improving flow in the DevOps value stream by analyzing data from the entire DevOps toolchain.

Differences Between DevOps VSMPs and DevOps VSDPs

VSMPs differ from VSDPs in the following ways:

■ Unlike VSDPs, VSMPs meet product teams where they are — i.e., acting as an orchestration
layer to the DevOps toolchain or the VSDP.
■ VSMPs provide visibility into higher-level metrics to enable product owners, application leaders
and I&O leaders to make decisions in an agile manner and to course-correct as needed. This is
by virtue of their integration with multiple data sources providing DevOps-related telemetry —
for example, the ability to inform and adjust release cadence, based on business demand and
users’ ability to absorb change.
■ VSMPs provide product teams flexibility and freedom of choice in the tools they want to use.
■ VSMPs have a strategic focus. VSDPs have an execution focus. And the two are not mutually
exclusive — VSMPs have API integrations with VSDPs to orchestrate actions (e.g., build, verify,
deploy, rollback) and provide visibility (see Figure 5).

Gartner, Inc. | G00464224 Page 9 of 16

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Figure 5. DevOps VSMPs

Critical Capabilities of a VSMP

VSMPs provide the following capabilities:

■ Flow metrics tracking and analysis — release velocity, team efficiency, lead time and cycle time
■ Integration with third-party DevOps tools — integrating with CI, CD, security and monitoring
tools
■ Workflow orchestration — create tickets, trigger application builds, deployments and rollbacks
■ Governance — enforce security and compliance controls as part of release management

Flow Metrics Analysis

VSMPs do not include native CI/CD capabilities, deferring the execution of the delivery pipeline to
VSDPs or other best-of-breed tools in the toolchain. However, integrations with those tools provide

Page 10 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 access to data to enable real-time analytics. The analytics helps identify bottlenecks, gaps and
redundancies across the DevOps value stream, as work progresses from one activity to the next.

Analytics enables product teams to realize multiple benefits:

■ CXOs, leadership teams and product owners acquire strategic views of product health.
■ DevOps is no longer just about “Dev” and “Ops”; it includes all stakeholders involved in the
delivery of customer value.
■ It enables and expedites data-driven decisions about future investments in the product.
■ The visualization capabilities help analyze customer-focused metrics, such as lead time,
deployment frequency, defect escape rate, feature adoption and time to respond to failures.

Integration With Third-Party DevOps Tools

Native/plug-in/prebuilt connectors to tools in the DevOps toolchain, including open-source tools

and other VSDPs. For example, integrate with Gitlab, CloudBees, GitHub, Jenkins, Jenkins X, Jira,
Selenium, JMeter, OWASP Dependency Check, Terraform, Spinnaker and others

Benefit: Minimizes the complexity of integrating multiple tools by providing prebuilt connectors.

Workflow Orchestration

Provides the ability to orchestrate actions, such as trigger builds, running automated tests,
application releases and automate change approvals.

Benefit: Allows product teams to continue using the tools with which they are familiar. In addition,
orchestration enables event-based pipelines and sequentially triggered pipelines.

Governance

VSMPs enable product teams to accomplish the following:

■ Enforce governance policies to establish roles, privileges and permissions that grant/deny
access to different pipeline activities
■ Set governance policies and continuously monitor and mitigate risk, without burdensome
administrative controls that impede agility
■ Adhere to regulatory compliance requirements mandated by internal and external audit
committees by establishing measurable governance parameters, such as peer reviews, access
controls and audit trails for all actions performed in the DevOps pipeline

Benefit: Minimize deviation from expected behavior that could otherwise impede quality, reliability
and security. Product teams can tailor the definition of “Done” based on the level of risk by
enforcing audit controls and automated change approvals. The change approvals can be based on
peer reviews and degree of test automation and code coverage.

Gartner, Inc. | G00464224 Page 11 of 16

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Best-of-Breed DevOps Tools Support Niche and Emerging Use Cases
Access to open source, adoption of cloud-native architectures and ML toolkits offer an opportunity
for technology-led innovation. This includes the need to support platforms and tools that allow
product teams to consume innovative technologies. Best-of-breed tools support these use cases.
As technologies become mainstream, VSDPs will provide native capabilities to support them.
Therefore, organizations must continuously assess the use of best-of-breed tools.

Five use cases prompt organizations to use best-of-breed tools:

■ Cloud-native application architectures

■ Security and compliance automation tools (to protect against new threat vectors)
■ Tools to operationalize ML operations (MLOps)
■ Open-source software (OSS) management
■ Commercial off-the-shelf (COTS) application management

Cloud-Native Tools
Cloud-native applications introduce the need for new platform capabilities — for example,
serverless applications in the cloud require new tools for granular and real-time observability.

Cool vendors have emerged to support deep profiling of serverless functions using fine-grained
telemetry data (see “Cool Vendors in Performance Analysis”).

Security/Compliance Tools to Protect Against New Threat Vectors

As product teams adopt new application architectures and development models, there is a need to
integrate security into the DevOps toolchain that supports those applications. For example,
because containers are fundamentally immutable, the need to scan container images upfront
requires specialized container-scanning tools for vulnerabilities. Comprehensive container security
starts in development with an assessment of the contents of the container, secrets management
and should extend into production with runtime container threat protection and access control (see
“Hype Cycle for Application Security”).

Tools to Operationalize MLOps

Building ML-enabled applications adds an additional layer of complexity, because of the
interdependence between training data and ML models. Monitoring the exact ML model in
production requires a combination of traditional application performance monitoring (APM)
approaches with real-time observations for model-specific metrics. Specialized DevOps tools
automate the process of packaging ML models with their associated dependencies into containers
using model-serving frameworks and link them to a CI/CD process.

Page 12 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 For example, Kubeflow and Polyaxon are designed to make deployments of ML workflows on
Kubernetes simple, portable and scalable (see “Artificial Intelligence Architect: A Key Role to
Operationalize and Scale Your AI Initiatives”).

Open-Source Software Management Tools

A DevOps culture provides fertile ground for the adoption of OSS tools. However, lack of proper
OSS governance exposes organizations to legal and security risks. This is a result of potential OSS
license violations and unpatched security vulnerabilities. Software composition analysis (SCA) tools
aim to mitigate these risks. SCA tools automate OSS governance by providing a detailed inventory
of components for the application (see “Four Steps to Adopt Open-Source Software as Part of the
DevOps Toolchain”).

COTS Application Management Tools

COTS presents a challenge to DevOps, in that managing COTS workflows tend to become
specialized, sometimes requiring proprietary tools. Common examples of COTS include ERP
systems, SaaS applications and core banking systems. Each has distinct challenges that make it
difficult to manage using standardized DevOps toolchains or DevOps value stream delivery
platforms. For example, using version control systems to manage custom code changes is a
standard best practice. However, as in the case of SaaS applications — custom code changes must
remain compatible and interoperable with “evergreen” SaaS environments, as SaaS providers
constantly upgrade their applications.

Conclusion
As outlined in this research, there are three ways that the DevOps toolchains will evolve. However,
these three ways are not mutually exclusive. Organizations will use VSDPs for a few products, layer
an existing DevOps toolchain with a VSMP, and use specialized tools to support emerging use
cases involving ML, cloud-native and open-source technologies.

Gartner Recommended Reading

Some documents may not be available as part of your current Gartner subscription.

“How to Build and Evolve Your DevOps Toolchains”

“3 Steps to Ensure Compliance and Audit Success With DevOps”

“Four Steps to Adopt Open-Source Software as Part of the DevOps Toolchain”

“Four Steps to Adopt Open-Source Software as Part of Your Test Automation Stack”

“Solution Path for Achieving Continuous Delivery With Agile and DevOps”

“Extending Agile With DevOps to Enable Continuous Delivery”

Gartner, Inc. | G00464224 Page 13 of 16

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Evidence
1 Gartner’s 2019 DevOps Survey

Results presented are based on a Gartner study conducted to assess the objectives, performance
and challenges faced in DevOps initiatives. It also delves into the performance, drivers and
challenges of scaling DevOps. The primary research was conducted online from 14 November 2018
through 18 December 2018, among 273 respondents in North America, Western Europe and the
Asia/Pacific (APAC) region.

Qualifying organizations span various industries, except “Services.” Companies were screened for
having annual revenue for FY17 greater than or equal to $100 million and to have minimum of 50
full-time IT employees. Companies were required to have DevOps to support systems/IT products in
production. DevOps efforts were required to be completely in-house or a mix of in-house and
outsourcing, with a minimum of five DevOps teams to support systems/IT products in production.
The sample represents organizations in the U.S. (n = 83), Canada (n = 35), the U.K. (n = 44),
Germany (n = 31), India (n = 48) and Australia/New Zealand (n = 32).

Respondents were required to have a role that is primarily IT-focused or be a fairly even blend of
business and IT. They were also required to be involved in decisions regarding DevOps efforts at
their organizations.

Quotas were applied for countries, industries and annual revenue.

The study was developed collaboratively by Gartner analysts and the Primary Research team.

Disclaimer: Results do not represent “global” findings or the market as a whole, but reflect the
sentiments of the respondents and companies surveyed.

2 Source of Social Media Analytics Data

Automated social media listening tools were used to track users’ responses on social media and
public discussion forums. The time period for the analysis was from 1 October 2017 through 28 July
2019. “Social media mentions” or “conversations volume” denote the inclusion of a monitored
keyword in a textual post on a social media platform. High counts of mentions should not be
considered an indication of “positive sentiment” or a measure of adoption by default.

User responses from social media were cleaned and segregated into different “Topics,” using the
Latent Dirichlet Allocation (LDA) method of topic modeling. This is based on the principle that each
Topic consists of a cluster of words that have high probability of appearing together, implying a
distinct “Theme.” The relevant topics that scored over a mean dispersion value were chosen and
labeled for both Theme and Geographical Analysis.

Social media sources considered for this analysis included Twitter, Facebook (publicly available
information only), Instagram, images (comments only), aggregator websites, blogs, news,
mainstream media, forums and videos (comments only). All geographical regions of the world were
analyzed for this study, except China. The social media data here is nonrepresentative of China, due
to restrictions imposed by the country on foreign-owned social media platforms.

Page 14 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 Social media analytics study results are not “market representative,” but largely “indicative.” They
reflect the aggregate crowdsourced opinion about a topic on social media.

Additional research contributions were provided by Ayush Saxena and Vidita Menon from the
Gartner Social Media Analytics team.

Gartner, Inc. | G00464224 Page 15 of 16

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.

 GARTNER HEADQUARTERS

Corporate Headquarters
56 Top Gallant Road
Stamford, CT 06902-7700
USA
+1 203 964 0096

Regional Headquarters
AUSTRALIA
BRAZIL
JAPAN
UNITED KINGDOM

For a complete list of worldwide locations,

visit http://www.gartner.com/technology/about.jsp

© 2020 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. This
publication may not be reproduced or distributed in any form without Gartner's prior written permission. It consists of the opinions of
Gartner's research organization, which should not be construed as statements of fact. While the information contained in this publication
has been obtained from sources believed to be reliable, Gartner disclaims all warranties as to the accuracy, completeness or adequacy of
such information. Although Gartner research may address legal and financial issues, Gartner does not provide legal or investment advice
and its research should not be construed or used as such. Your access and use of this publication are governed by Gartner Usage Policy.
Gartner prides itself on its reputation for independence and objectivity. Its research is produced independently by its research
organization without input or influence from any third party. For further information, see "Guiding Principles on Independence and
Objectivity."

Page 16 of 16 Gartner, Inc. | G00464224

This research note is restricted to the personal use of rodrigues.kwate-kwate@hec.ca.