Professional Documents
Culture Documents
From Monitoring
to Observability
Dave McAllister
Sr. Technical Evangelist
Stephane Estevez
Product Marketing Director EMEA, IT Markets
Forward- During the course of this presentation, we may make forward‐looking statements regarding
future events or plans of the company. We caution you that such statements reflect our
Looking current expectations and estimates based on factors currently known to us and that actual
events or results may differ materially. The forward-looking statements made in the this
Statements presentation are being made as of the time and date of its live presentation. If reviewed after
its live presentation, it may not contain current or accurate information. We do not assume
any obligation to update any forward‐looking statements made herein.
In addition, any information about our roadmap outlines our general product direction and is
subject to change at any time without notice. It is for informational purposes only, and shall
not be incorporated into any contract or other commitment. Splunk undertakes no obligation
either to develop the features or functionalities described or to include any such feature or
functionality in a future release.
Splunk, Splunk>, Data-to-Everything, D2E and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States
and other countries. All other brand names, product names or trademarks belong to their respective owners. © 2020 Splunk Inc. All rights reserved
© 2020 SPLUNK INC.
SignalFx
Infrastructure Monitoring is now Splunk Infrastructure Monitoring
SignalFx
Microservices APM is now Splunk APM
© 2020 SPLUNK INC.
Known Unknown
Things we are aware of Things we are aware of but
AND understand DON’T understand
Known
Known Unknown
Things we are aware of Things we are aware of but
AND understand DON’T understand
Known Monitoring
Known Unknown
Things we are aware of Things we are aware of but
AND understand DON’T understand
Known Monitoring
Monitoring Observability
Looking for expected problems, e.g.: Looking for new and missing
• Applications data to enhance monitoring
• Overloaded CPU • Existing Environments
• High Memory Utilization • Containers
• Disk Space • Serverless
• High Response Latency • Microservices
• High Error Rate • Multi-clouds
• Service Availability • Anything else that can fail, but hasn’t (yet)
© 2020 SPLUNK INC.
Analyze
Monitoring
A Verb
Something you do to determine the state of
an application, a system, a service…
Observability
A Noun
A thing you have –
a property of a system
© 2020 SPLUNK INC.
Analyze
Monitoring
A Verb
Something you do to determine the state of
an application, a system, a service…
Observability
If you are observable A Noun
A thing you have –
a property of a system
© 2020 SPLUNK INC.
Analyze
Monitoring
A Verb
Something you do to determine the state of
an application, a system, a service…
Observability
If you are observable A Noun
A thing you have –
a property of a system
I can monitor you
© 2020 SPLUNK INC.
Analyze
Monitoring
A Verb
Something you do to determine the state of
an application, a system, a service…
Observability
If you are observable A Noun
A thing you have –
a property of a system
I can monitor you
find patterns
© 2020 SPLUNK INC.
Analyze
Monitoring
A Verb
Something you do to determine the state of
an application, a system, a service…
Observability
If you are observable A Noun
A thing you have –
a property of a system
I can monitor you
find patterns
Source: Wikipedia
© 2020 SPLUNK INC.
Source: Wikipedia
© 2020 SPLUNK INC.
Observability
The Three Pillars
Observability
The Three Pillars
Observability
The Three Pillars
Observability
The Three Pillars
Observability Drives
Evidence-based Debugging
AIOps
Monitoring / Alerting
I monitor you
VM VM VM VM VM VM
Private Public
© 2020 SPLUNK INC.
AIOps
Monitoring / Alerting
I am observable
LOGS METRICS TRACES
VM VM VM VM VM VM
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
© 2020 SPLUNK INC.
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
1 1
1
Metrics
0
0
%
0 0
Ops
0 0
% %
Monitoring 0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
© 2020 SPLUNK INC.
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
1 1
1
Metrics
0
0
%
0 0
Ops
0 0
% %
Monitoring 0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
Sessions
Monitoring Business
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
© 2020 SPLUNK INC.
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
1 1
1
Metrics
0
0
%
0 0
Ops
0 0
% %
Monitoring 0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
Sessions
Monitoring Business
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
DB Monitoring DBA
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Service
Change mgnt
8 9 1 1 1 1 2 3 8 9 1 1 1 1 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
Mgnt
2
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Problem
Log analysis
Mgnt
© 2020 SPLUNK INC.
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
1 1
1
Metrics
0
0
%
0 0
Ops
0 0
% %
Monitoring 0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
Sessions
Monitoring Business
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
DB Monitoring DBA
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
© 2020 SPLUNK INC.
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
1 1
1
Metrics
0
0
%
0 0
Ops
0 0
% %
Monitoring 0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
Sessions
Monitoring Business
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
DB Monitoring DBA
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Service
Change mgnt
8 9 1 1 1 1 2 3 8 9 1 1 1 1 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
Mgnt
2
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Problem
Log analysis
Mgnt
© 2020 SPLUNK INC.
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
1 1
1
Metrics
0
0
%
0 0
Ops
0 0
% %
Monitoring 0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
Sessions
Monitoring Business
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
DB Monitoring DBA
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Service
Change mgnt
8 9 1 1 1 1 2 3 8 9 1 1 1 1 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
Mgnt
2
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Problem
Log analysis
Mgnt
© 2020 SPLUNK INC.
Tracing
Tracing Dev
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
1 1
1
Metrics
0
0
%
0 0
Ops
0 0
% %
Monitoring 0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
0
%
8
A
M
9
A
M
1
0
A
M
1
1
A
M
1
2
P
M
1
P
M
2
P
M
3
P
M
Sessions
Monitoring Business
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
DB Monitoring DBA
8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Service
Change mgnt
8 9 1 1 1 1 2 3 8 9 1 1 1 1 3 8 9 1 1 1 1 2 3 8 9 1 1 1 1 2 3
Mgnt
2
A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P A A 0 1 2 P P P
M M A A P M M M M M A A P M M M M M A A P M M M M M A A P M M M
M M M M M M M M M M M M
Problem
Log analysis
Mgnt
© 2020 SPLUNK INC.
External (customer’s)
view is singular
• Request, and its latency
and success
Why Now?
© 2020 SPLUNK INC.
Tightly Coupled Apps, Primarily using More Modular, but Loosely Coupled
Slow Deployment Cycles Cloud IaaS Dependent App Microservices, and
Components Serverless Functions
VM VM VM VM VM VM VM VM VM
VM VM VM VM VM VM
What’s Different?
Cloud-native boosts velocity, but also increases complexity
Elastic,
Complex Short-Lived “You Build It,
Interdependencies Infrastructure You Run It”
• 10s or even 100s of loosely • Multi-cloud, abstracted • Monitoring is not limited to Ops –
coupled, polyglot services infrastructure is extremely developers are key users
• System behavior is dynamic • No single user has an accurate
unpredictable and changes • Volume of objects and mental model – troubleshooting is
over time metrics to monitor skyrockets a team sport
© 2020 SPLUNK INC.
Unlimited Cardinality
Data Collection
No cardinality limits
Data
Visualization
• Monitoring, Analytics,
Response tooling
• OOTB and customizable
dashboards
• Real-time feeds
• Real-time smart alerting
• No lost data
• And more
• Unlimited cardinality metrics • Founder & leading contributor • AI-driven directed troubleshooting
• NoSample™ full-fidelity traces • Intelligent & automated response
• No schema, streaming logs
© 2020 SPLUNK INC.
Open Instrumentation
Metrics Agents | Cloud APIs | Function Wrappers | Tracing Auto-Instrumentation
OBSERVABILITY
All Data, Any Source, Any Scale
Unlimited cardinality metrics, NoSample™ full-fidelity Immediate Detection
traces and unstructured logs from infrastructure, Discover unknown unknowns, catch all
apps/services and business processes outliers and anomalies, alert in seconds
© 2020 SPLUNK INC.
SESSION SURVEY