Observability Maturity

Community Research Findings 2021

Table of Contents

Executive summary 3

Key findings 5

The observability spectrum 10

The benefits of practicing observability 14

Test your observability capabilities 18

Recommendations and resources 19

Conclusion 20

Observability Maturity Community Research Findings 2

Executive summary
The 2021 Observability Maturity Community Research Findings report is a follow-
up to the inaugural report published in Q1 2020. The survey was commissioned by
Honeycomb, which provides full-stack observability designed for high-cardinality
data and collaborative problem-solving. Founded by Christine Yen and Charity
Majors, two engineers with experience debugging problems at scale for tens of
millions of users, Honeycomb empowers every engineer to interrogate application
telemetry data in granular and arbitrary ways, so they can deeply understand and
debug the behavior of their production systems.

The survey was conducted by ClearPath Strategies, with the goal of

understanding community perceptions and awareness of observability, how
engineering teams are approaching observability, and mapping an observability
maturity scale that updates last year’s findings.

The 2021 survey found that, overall, observability adoption is on the rise.
Organizations on the higher end of the observability maturity spectrum are
realizing benefits, such as:

• Higher productivity
• Improvement in code quality
• End-user satisfaction
• Software developer retention

Observability Maturity Community Research Findings 3

While overall adoption is on the rise, the report also finds that a majority of teams
are at the earliest stages of observability maturity. Teams on the lower end of the
spectrum are not realizing the same outcomes we see from organizations at the
higher end, such as deploying more frequently, finding bugs more quickly before
and after pushing to production, and reduced burnout.

Teams further along the observability maturity spectrum not only realized
additional benefits; they also realized those benefits with higher-quality
business outcomes—for example, virtually all the Advanced and nearly 9 in 10
of Intermediate respondents provide customer experiences that leave their
customers either “Always Satisfied” or “Sometimes Satisfied” with their services.

In short, teams on the higher end of the observability spectrum realize clear value
in several areas, including an improved ability to identify problems and solve
those problems faster, more satisfied customers, and more confidence deploying
frequently to production when compared to their less-mature counterparts.
The 2021 report details how observability can help your organization achieve
production excellence and gain a competitive edge.

This survey was conducted from December 9, 2020 – January 11, 2021. Invites to participate in the
survey were distributed via a Honeycomb email list and social media outreach. A total of 405 respondents
participated in the survey. While the margin of sampling error cannot technically be calculated for online
populations where the relationship between sample and universe is unknown, the margin of sampling error
for equivalent representative samples would be +/- 4.9%.

Observability Maturity Community Research Findings 4

Key findings
The second wave of our Observability Maturity Community Research Findings
verifies that observability adoption is on the rise, with more survey participants
practicing observability today than they were in the 2020 report. However, there
are signs that confusion may exist around how observability is implemented,
especially among groups lower on the maturity spectrum. Teams that score
lower on the maturity spectrum report less overall comprehensive understanding
of their production systems and also realize fewer benefits than their more
mature counterparts.

While observability adoption increased this year, shifts in maturity were only
significant at the upper and middle parts of the spectrum (a 4% increase in
Advanced teams and a 2% increase in Intermediate teams). Respondents
indicated several barriers to progressing their goals, but those in the middle of
the maturity spectrum disproportionately cited lack of skills to implement as a
barrier to adoption.

Key findings for the 2021 report include:

01 Observability is gaining traction. More organizations are practicing

observability today than in the 2020 report. Sixty-one percent of
respondents reported that their teams are currently practicing
observability, an increase of 8% from last year. That increase is
sharply reflected across individual teams (up 7%) as opposed to entire
organizations (up only 1%).

Observability Maturity Community Research Findings 5

 02 Teams on the higher end of the maturity spectrum realize more
benefits than their less-mature counterparts. Teams are attracted to
observability’s promise to help them understand problems better and
solve problems faster—and teams that are mature in their observability
practice realize even more impactful business outcomes, including
deploying more frequently, being able to find bugs more quickly before
and after pushing to production, and reduced burnout.

03 More-mature teams are also 3X more likely to deliver higher customer

satisfaction. Teams that have achieved Intermediate or Advanced-level
maturity reported their end-user customers are “Always Satisfied” with
their service quality and capability at a rate of three times more than
teams that do not practice observability.

04 Lack of implementation skills is a disproportionate barrier for

observability adoption. While interest in observability has gained
significant momentum, some organizations face barriers to reaching
higher levels of maturity, such as lack of implementation skills, difficulty
scaling to other teams, and competing with other initiatives.

Observability Maturity Community Research Findings 6

Observability adoption is primarily being driven by software developers, followed
closely by DevOps engineers, and site reliability engineers (SREs). These groups
are driving interest in observability, both in terms of introducing observability to
their organizations and persuading decision-makers to implement the practice.
However, this grassroots adoption pattern results in most teams who planned
to adopt observability doing so on a team-by-team basis, rather than driving
adoption across the entire organization.

Fig. 1

Devops/SRE

Developers

Architects

Managers

CIO/CTO

Operations

Figure 1: When asked which members of their team were responsible for driving decisions to practice
observability, respondents primarily chose Developers, DevOps, and SREs. Some respondents identified
as the person in their team who introduced observability practices; their roles were 21% developers, 36%
DevOps or SRE, 14% managers, 16% architects, 10% CIO/CTO, and 3% operations.

Observability Maturity Community Research Findings 7

The teams that have adopted observability practices reported increased
confidence in catching bugs before and after production, in addition to a number
of other benefits, including the ability to:

• Understand what’s happening inside their system at any time, without

shipping new code (70% of respondents)

• Immediately identify the problem when something breaks and the impact it
has on other systems (69%)

• Understand their entire system, from high-level trends to specific events and
outliers (63%)

• Immediately identify the solution to a problem (51%)

Fig. 2

High maturity correlates to more comprehensive Advanced/Intermediate

Novice
understanding of the entire system Planning
No Plans

Figure 2: Respondents were asked to indicate whether they agreed or disagreed that their teams had
various capabilities that gauge a comprehensive understanding of their production systems.

Observability Maturity Community Research Findings 8

However, while interest in observability has gained significant momentum,
we also found that maturity is shifting at different paces: The Intermediate
and Advanced groups have shifted toward higher maturity over the past year.
Overall, the proportion of these two higher-spectrum groups remained constant.
This could suggest that groups who are well-versed in observability practices
are accelerating their skills and may be pulling away from their lower-maturity
counterparts.

In contrast, the Novice group has remained stagnant year-over-year, despite

more respondents reporting that they have started their observability adoption
journey. We would have expected to see a shift from the Novice group toward the
Intermediate group if progression through the maturity curve was happening at a
consistent pace. The stagnation suggests that some groups may get stuck in the
middle of their adoption journey. When asked about adoption barriers, the Novice
group disproportionately cites lack of implementation skills in comparison to all
other groups. (See Figure 5.)

Most respondents fall in the evolving middle, where they practice observability
processes or tooling, but rarely both. They also report some, but not most, key
capabilities, such as identifying and resolving bugs before and after deploying
to production. Lastly, at the time of the survey, approximately one in five
respondents were not practicing or using observability tooling, but have plans to
do so within the next year.

Observability Maturity Community Research Findings 9

The observability spectrum
Observability is measured as a set of capabilities that help engineering teams
gain comprehensive understandings of their production systems. Those
capabilities are sociotechnical in nature—they have a technical component, as
well as a component of social culture and practice. Our observability maturity
model reflects the ongoing nature of the continuous learning and improvement
needed to unlock these capabilities. This year’s survey found that most
respondents have adopted some, but not all, of the practices required to achieve
the biggest benefits associated with observability.

Our observability maturity model evaluates the extent to which teams practice
and realize the benefits of observability, and scores them along an observability
maturity spectrum. The observability maturity spectrum identifies five distinct
groups:

1. No Plans. Those who do not have plans to adopt observability or whose

behaviors and capabilities suggest barriers to adoption.

2. Planning. Those who indicate an intention to practice observability in the next

1–2 years and whose behaviors and capabilities suggest their organization is
well positioned to succeed in observability adoption.

3. Novice. These respondents report practicing observability, but their practices

and processes suggest they are in the earliest stages of adoption.

4. Intermediate. Respondents who are practicing observability and are beginning

to achieve a comprehensive understanding of their production systems.

Observability Maturity Community Research Findings 10

5. Advanced. Respondents who are practicing observability and realizing both
comprehensive understandings of their production systems and benefits
associated with that practice.

Fig. 3

Figure 3: This figure shows a spectrum of observability maturity that correlates with how well teams
report they are able to demonstrate five key capabilities.

Advanced and Intermediate

In the Advanced and Intermediate groups, over 10% of those surveyed reported
a combination of practices and tooling that reflect a highly observable system.
Another 13% of the Intermediate respondents reported practices, tooling, and
outcomes consistent with relatively sophisticated observability practices,
down from 17% last year—but that 4% shifted to the Advanced group on the
observability spectrum. In other words, we’re seeing groups higher on the
maturity scale continue to improve.

These two groups highly prioritize observability: 50% practice observability

across the organization and 43% on a team-by-team basis. Respondents also
reported high public cloud use and were from a mix of company sizes, but most
work at large enterprises (57%) and in the tech industry (46%).

Observability Maturity Community Research Findings 11

Novice

Most survey respondents fall into the Novice group. Participants report
observability processes or tooling, but rarely both. There may be several reasons
for this misalignment, but we believe this may reflect mismatched expectations
among practitioners in this group.

Analyzing responses, this group is more likely to self-report that they are
practicing observability because they are using tools like logs, metrics, and
traces. However, they also do not report having the key capabilities associated
with an observability practice. They report that they do not have a comprehensive
understanding of their systems. They also report some, but not most, key
capabilities. Additionally, one in two respondents in this group indicated that
practices like observability and DevOps are siloed within their organization or
only practiced on a team-by-team basis.

Planning

Teams in this group reported practices and tooling that suggest identifying and
resolving problems faster are top priorities. More specifically, in this group, one
in four are at the very beginning of their observability journey and are starting
to practice on a team-by-team basis (26%), while one in three respondents plan
to practice observability in the next 12 months (33%). Approximately one in five
respondents do not currently practice or use observability tooling, but have plans
to do so within the next year.

Observability Maturity Community Research Findings 12

No Plans

Seventeen percent of survey respondents reported that they do not practice

observability, do not use observability tools, and have no plans in the near future
to adopt. Respondents in this category help provide baselines for expectations
and performance when compared to the other groups.

Of the 17% of respondents that are unpracticed, more than half are enterprise-
sized (52% with 1,000+ employees) financial services (22% finserv) companies
and report higher Azure use (39%).

Fig. 4

Our Target Audiences

Figure 4: This year, we saw a shift from Intermediate to Advanced as well as from Planning to Novice.
However, no similar shift was seen from Novice to intermediate, suggesting that adoption barriers may
be stalling progress for Novice teams.

Observability Maturity Community Research Findings 13

The benefits of practicing observability
Respondents are looking to observability to help them identify and resolve
problems faster, and in turn, help them create more resilient systems. Our year-
over-year research verifies that teams on the higher end of the maturity spectrum
are more likely to have:

• Code that is well understood, well maintained, and fewer bugs than average

• The ability to follow predictable release cycles because they confidently

address issues that arise

• Understanding of the end-to-end performance of their systems and how

technical debt is costing their organization

• The ability to visualize context-rich events that allow efficient, focused, and
actionable on-call processes

• The ability to prioritize responsiveness to user behavior and feedback

• Completely automated or mostly automated releases, resulting in reduced toil

• The ability to set and measure service level objectives, resulting in better
alignment between engineering and business goals

Observability Maturity Community Research Findings 14

 Fig. 5

Understanding the barriers teams face Novice

Total

Figure 5: Groups in the Novice stage of maturity disproportionately report lack of implementation
skills and too much technical debt as barriers to adoption.

Fig. 6

Maturity increases confidences in catching bugs BEFORE production

Advanced/Intermediate
Novice
Planning
No Plans

Observability Maturity Community Research Findings 15

 Figure 6: 81% of Advanced and Intermediate teams report confidence in catching bugs before
production vs. 1% of teams with No Plans. 77% of No Plans teams report little to no confidence
catching bugs before production.

Fig. 7

And after production

Advanced/Intermediate
Novice
Planning
No Plans

Figure 7: 97% of Advanced and Intermediate teams report confidence in catching bugs before
production vs. 7% of teams with No Plans. 58% of No Plans teams report little to no confidence
catching bugs before production.

Teams and organizations that are further along the observability maturity
spectrum realize better business outcomes, including more satisfied customers.
High maturity correlates to a more comprehensive understanding of the
entire system, which increases confidence in catching bugs before and after
production.

Observability Maturity Community Research Findings 16

 Fig. 8

Observability Maturity Leads to Better Advanced/Intermediate

Novice
Business Outcomes Planning
No Plans

Figure 8: Intermediate & Advanced teams are three times more likely than teams who have not started
down the path of observability, and twice as likely as Novice teams, to deliver customer experiences
that always satisfy customers.

Observability Maturity Community Research Findings 17

Test your observability capabilities
Wondering how your organization scores on the observability maturity spectrum?
Answer these questions to find out.

1. Does your organization understand the state of your entire system from end
to end, at any given time?

2. Can you identify the source of problems in production without shipping new
code?

3. When something breaks, do you have the ability to understand who is being
impacted?

4. How confident are you in your team’s ability to catch bugs BEFORE being
released to production?

5. How confident are you in your team’s ability to catch bugs AFTER being
released to production?

Observability Maturity Community Research Findings 18

Recommendations and resources
To help teams unfamiliar with observability or who are just getting started on
their journey get better acquainted with observability practices and the benefits,
we recommend the following resources.

• Accelerate: The Science of Lean Software and DevOps: Building and Scaling
High Performing Technology Organizations, by Nicole Forsgren, PhD; Jez
Humble, and Gene Kim

• Site Reliability Engineering, by Betsy Beyer, Chris Jones, Niall Richard Murphy,
and Jennifer Petoff

• 2019 Accelerate State of DevOps Report (DORA)

• Guide to Achieving Observability - Honeycomb

• OpenTelemetry Project

• Follow Charity Majors and Liz Fong-Jones on Twitter

• Charity Majors’ personal blog

Observability Maturity Community Research Findings 19

Conclusion
Observable systems and practices enable production excellence. Our combined
decades of field experience have witnessed it firsthand by working directly with
software engineering teams of all sizes. These research findings confirm our
qualitative insights: Teams practicing observability are realizing undeniable
value in their improved ability to identify problems and solve those problems
faster.

Observability is a still-nascent and rapidly evolving field. Organizations with

teams on the journey—especially those in the Novice and Planning stages—are
seeking guidance and best practices. While there is no one-size-fits-all approach
to observability or a specific starting place, teams on the higher end of the
maturity spectrum share outcomes that underpin production excellence.

As you determine your current place on the observability spectrum, we

recommend conducting upfront research to fully understand the landscape of
tooling and all the available options. But do not focus on only tooling or internal
practices—instead, adopting them both together will support and reinforce your
investment and payoff.

Finally, take the time to bring all software engineering team members together
and practice observability on a daily basis in order to realize the meaningful
benefits. Adopting a culture of shared ownership during incident response and
post-incident reviews will help address system-wide issues and also reduce risk

Observability Maturity Community Research Findings 20

to service performance and impact on end users. Allow time to adopt and learn
the necessary skills, and the payoff will be significant—faster incident response,
more performant digital services, faster and more reliable feature releases
to production, and, most importantly, happy engineering teams and happy
customers.

Observability Maturity Community Research Findings 21