© 2019 SPLUNK INC.

Kubernetes
Observability with
Splunk Connect for
Kubernetes (SCK)

October 2019
 © 2019 SPLUNK INC.

Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward-looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved.
 © 2019 SPLUNK INC.

Donald Tregonning Shubham Jain

Senior Software Engineer | Splunk Senior Software Engineer | Splunk
Chaitanya Phalak
Software Engineer | Splunk
 © 2019 SPLUNK INC.

Demo
Sneak peek into today’s presentation
 © 2019 SPLUNK INC.

1. Environment Overview – Splunk & K8s

2. Observability Challenges & Problems solved

3. Splunk Connect for K8s (Intro, Design and Architecture)

Agenda
4. SCK Installation and Configuration

5. Kubernetes Metrics and Splunk Metrics Workspace

6. Advanced Splunk Tooling with SCK

7. SCK Highlights and Roadmap

8. Q&A
 © 2019 SPLUNK INC.

Splunk & SCK Environment

 © 2019 SPLUNK INC.

Challenges with Observability

▶ 3 pillars of observability
Logs
▶ Loosely tied together

▶ Resides in different systems

Observability

Trace
Metrics
s
 © 2019 SPLUNK INC.

Splunk Connect for Kubernetes brings

together all three pillars into
one solution…
 © 2019 SPLUNK INC.

Problems tackled with SCK

✔ Tackled Heapster deprecation and removal from K8s Project

✔ Options to build out metrics
• Prometheus
• Simple Summary API Client
• Custom Metrics solution (Winner)
✔ Reliability
✔ Metric Store Compatibility
✔ Elasticity
✔ Version Support
✔ Scalable
 © 2019 SPLUNK INC.

Different Types of
Data in Splunk

▶ Metric data
▶ Event data
• Log data
• Objects metadata
 © 2019 SPLUNK INC.

Splunk Connect for

Kubernetes Internals
An Introduction, Design Patterns and Product Architecture
https://github.com/splunk/splunk-connect-for-kubernetes
 © 2019 SPLUNK INC.

SCK Component
Component types
• Logs
• Metrics
• Objects
Collector agent: Fluentd
• Fluentd plugins for component
functionality
Deployment types
• K8s Deployment
• K8s Daemonset
 © 2019 SPLUNK INC.

Logging Component
Deployment - daemonset
• configmap
• secret
• serviceAccount
Fluentd plugins
• in_tail
• systemd
• monitor_agent
• concat
• jq_transformer
Logs are sent to Splunk using Splunk’s
• splunk_hec
fluentd HEC plugin (splunk_hec)
 © 2019 SPLUNK INC.

Metrics Component
Deployments
• daemonset
• Supporting kubernetes objects
(configmaps, secrets and
serviceAccounts)

Fluentd plugins
• kubernetes_metrics
• kubernetes_metrics_aggregator
• splunk_hec
Metrics are sent to Splunk using Splunk’s
fluentd HEC plugin (splunk_hec)
 © 2019 SPLUNK INC.

Objects Component
Deployment
• configmap
• secret
• serviceAccount
Fluentd plugins
• kubernetes_objects
• splunk_hec
Objects plugin
• Cluster’s events metadata for
configured kubernetes objects (pods,
nodes, configmaps)
Collected events are sent to Splunk using
Splunk’s fluentd HEC plugin(splunk_hec)
 © 2019 SPLUNK INC.

Installation and
Configuration
How to install and configure Splunk Connect for K8s
using Helm
 © 2019 SPLUNK INC.

Splunk Prerequisites
What you need to get started

​SplunkEnterprise 7.0 or ​SplunkHTTP Event ​Two Splunk indexes

later version Collector ready
 © 2019 SPLUNK INC.

Kubernetes Prerequisites
What you need to get started

​Helm to deploy SCK ​Admin

access to ​Ability
to install SCK
Kubernetes cluster to run using yaml manifests
helm
 © 2019 SPLUNK INC.

Demo
Splunk Connect for Kubernetes
 © 2019 SPLUNK INC.

K8s Metrics
Endpoints and Splunk
Metrics Workspace.
Where we collect things and where they end up
 © 2019 SPLUNK INC.

Metrics Collected in SCK:

Where do we https://github.com/splunk/fluent-plugin-kubernetes-metrics/blob/develop/metrics-information.md

collect K8s Metrics Collection Endpoints:

Metrics from? 1.) @kubelet_url = "https://#{ env_host}:#{env_port}/stats/summary"

2.) @kubelet_url_stats = "https://#{ env_host}:#{env_port}/stats/"

3.) @cadvisor_url = "https://#{ env_host}:#{env_port}/metrics/cadvisor"

4.) @kubernetes_url_final = "https://#{ env_host}:#{env_port}/api/"

 © 2019 SPLUNK INC.

Splunk Metrics Workspace

 © 2019 SPLUNK INC.

Advanced Splunk
Tooling with SCK
Introducing Splunk App for Infrastructure with SCK
 © 2019 SPLUNK INC.

Splunk App for Infrastructure

Easily deploy Splunk Connect for Kubernetes and monitor your k8 environment

• Unifies and correlates logs

and metrics for
comprehensive
infrastructure monitoring
and troubleshooting
• Guided, out-of-the-box
data onboarding to monitor
Kubernetes performance
and health
• Identify trends and find
root causes fast
 © 2019 SPLUNK INC.

Demo
Splunk App for Infrastructure
 © 2019 SPLUNK INC.

Splunk Connect
Kubernetes
Highlights and What’s
Next
Next Release – Now!
 © 2019 SPLUNK INC.

1. Works well with all Kubernetes flavors (self-managed or hosted: EKS,

GKE, AKS, OpenShift and IBM IKS…and everything else)

2. Reliable event delivery leveraging Splunk’s HTTP Event Collector.

3. Scalable and secured way of ingesting k8s data into Splunk

SCK
Highlights 4. Ships with features like:
• Index routing
Product summary
• Data filtering
• Support for Custom Metadata
• Support for audit logs

5. Actively developed and maintained by Splunk and its community!!

6. Support for upcoming Splunk Data Ingestion product. Ingest API

7. Simple, performs exceptionally well

 © 2019 SPLUNK INC.

What’s next for the SCK!

Support for
PKS

Flexible
security
configurations
(PSP)
More metrics
alongside
improved
performance
Better out of
the box tracing
capability
 © 2019 SPLUNK INC.

Important Project Links & Contributions

Github repo links for different component of SCK

• https://github.com/splunk/splunk-connect-for-kubernetes
• https://github.com/splunk/fluent-plugin-kubernetes-objects
• https://github.com/splunk/fluent-plugin-splunk-hec
• https://github.com/splunk/fluent-plugin-kubernetes-metrics
• https://github.com/splunk/fluent-plugin-k8s-metrics-agg

▶ Contributions -
https://github.com/splunk/splunk-connect-for-kubernetes/issues/new
 © 2019 SPLUNK INC.

Thank
You!
Don’t forget to rate our session
 © 2019 SPLUNK INC.

Q&A
Come visit us at the GDI booth on the main floor!!