Enterprise K8S Logging

Entercloud

Andrew S Thompson

09.30.2018

Entercloud| 2310 s Canal street, Chicago, USA 60616 | www.entercloud.info

Backlog Topics

Overview................……….…….……………………… 3
Initial summary and environment assumptions


General Problems ……………….…….. 4
Logging resources - current state of enterprise


General Solutions ……………………………………. 5
Logging solutions - opensource technologies


Key introspection topics ……..……………………………. 6
Separate backends instantiated to store, analysis and query logs.


K8S Enterprise logging …………..………………………. 7
Consolidate into single access custom dashboard provide access to stack
holders as needed.

Summary and Flow….…………..………………………. 8

Log Flow overview and CE to EE migration


!2
Overview
Based on previous meetings from 09/24/18 - 9/30/18 the topic of backlog items:

✴Kubernetes and App Logging and monitoring

✴Migrate Docker CE to EE
✴ Secure Jenkins X CICD pipeline.

Compiled here is the use cases per topic/item

Code Snippets - https://github.com/andrewpsp/lab

Caveat: The project work was previously built on open source and trial software that has since expired. items where
originally presented as Proof of concepts and have since have pivoted to reference/base configuration work.

So, let’s get started!

!3
General logging problem..
Functionality provided by a container engine or runtime is usually not enough for a
complete logging solution. For example, if a container crashes, a pod is evicted, or a
node dies, you’ll usually still want to access your application’s logs. As such, logs should
have a separate storage and lifecycle independent of nodes, pods, or containers.

⁃ K8S provides no native solution for log data.

Initial requirements to solve…

Requires separate backend to:

• store
• analysis
• query logs

Monitoring is collecting, parsing, storing, visualizing, and acting on the logs and
metric data

Current state of monitoring and limitations

⁃ Kubelet keeps containers with logs - but if the pod is evicted all
corresponding containers and logs are also evicted

- Node-level logging - can easily consume all storage on node and is not set
by default but can be configured easily in the kube-up.sh

- Rotates logs every hour with minimum retention.

- Once the log is rotated only the contents of the latests log file will available
from the kubelet logs, container logs, system logs, and journald 


!4
General Logging Solution
Monitoring Enterprise system that automatically detect the changes and adapt to
them
⁃ Logs
⁃ Metrics
⁃ Monitoring deployments
⁃ Monitoring deployment scaling

Golden Signals are build the metric frameworks around:

✤ Proactive monitoring
✤ Cluster visibility and capacity planning
✤ Trigger alerts and notification
✤ Metrics dashboards

Logging topics has separate storage and separate lifecycle

independent of nodes, pods or containers

Cascading hierarchy of roles and bindings - make use of AggregateRole

‣ Namespaces per component / application
‣ RBAC per component / application

PodSecurityPolicy 1.12 Beta - Best Solution

This object define a set of conditions that a pod must run with in order to be
An accepted into the system

Collects, aggregates, processes, and exports information about

running containers and Automatically scaling or adapting the
cluster based on its current state
Explicitly spell out privileges , and attach ServiceAccount to the aemonSet.

https://github.com/andrewpsp/lab/tree/master/k8s_resources

Multiple Environments: Consolidate into single access custom dashboard

provide access to stack holders as needed.

!5
Key topics require dedicated introspection:
Separate backends instantiated to store, analysis and query logs.

Opinionated metrics pipeline solution based on Ceph(Rook)

could very easily apply to

• awsElasticBlockStore
• glusterfs
• rbd

- pods level consumption w/ Block storage

- Ingress level storage inside and outside the k8s cluster w/ Object storage
- pods share the same filesystem across the system w/ Shared Filesystem
- Ability to resize PVC as to provide the long term storage solutions and retention
policies

https://github.com/andrewpsp/lab/tree/master/k8s_resources/cloud_log

Solarwinds Flavored Fluentd DaemonSets

- Allows integration with Loggly and Papertrail
- Connects with the Elastic search cluster and extends the Kibana visualization layer

https://github.com/andrewpsp/lab/tree/master/k8s_resources/log_stack/solarwind_integration

Metrics Server and Kube-State

Direct support for Prometheus operators and exporters.

- Scraping endpoints
- pulls from audit policy objects
- Pulls from kube-state
- Pushes to alertmanager
https://github.com/andrewpsp/lab/tree/master/k8s_resources/alerts_stack

- Take existing kublet logs and logging agent already present on the node reads
logs from file, sockets and journald etcd,
- kube-scheduler or kube-controller can expose its internal performance metrics

!6
Enterprise k8s logger Objects:

Deployments
 DaemonSets, Rook Operator
Pods,Services,
StorageClass, Bindings,
ConfigMaps ,
namespaces, Endpoints
kube-apiserver API Management & Health

New Objects:

volume Rook
attachments Agent 
,Prometheus   
FlexDriver Ceph-
Volume  tools
S3
Attach & Mount Operations  

Overview:
Kubelet

metric-server
Rook Volume
CAdvisor  Plugin

Prometheus Fluentd
ServiceMonitor
ElasticSearch 
ServiceMonitor
Alertmanager Solarwind Integration

UI   various sources on
Alerts  resizable volume claims 
Various Pods
Alerts and Recording rules
namespace 1N

Grafana Custom
Event / Trigger
Kibana

Notification / Grafana
Actions / Notice
Namespace 2n

Action  Kibana

Custom

Kibana Custom
Namespace 3n

Grafana
Namespace 4n

Docker Trusted
Loggly cloudwatch
registry

Rook
Papertrail Dashboard

๏ Explicitly create services that will point to the component’s pod and namespaces to establish
profiles and baselines
๏ Watches externally or internally on paths from the Ingress controller
๏ Consolidate into single access custom dashboard provide access to stack holders as needed.

!7
Summary Image

Log Flows

With the proper permissions Rook uses FlexVolume to integrate with Kubernetes for performing
storage operations. This interface talks directly to the Rook API to provision. The API provides a
tighter control and a more integrated solution to provision volumes. Resizing persistent volume
claims on-premises storage solutions like Rook is not a big issue like other StorageClass
solutions. The part of the reason is that when you create a volume in Ceph, let's say a 100GB,
this space is not actually allocated.

Move from CE to EE via Multi-stage builds

Under the assumption that current Dockerfile is being built via CI and pushed to internal
repository. The migration implementation to update the source would add a post build step to
append the desired DTR image into the legacy Dockerfile and copy to contents into the new
image - this method is directly supported by docker since version 17.05

CE Dockerfile - Legacy

EE Dockerfile - Multi-stage builds

!8
 "First, solve the problem. Then, write the code." - John Johnson

Thank You

For more information, contact: Andrew S Thompson

admin@entercloud.info

TIME CONSTRAINTS HAVE LIMITED THIS SOLUTION HIGHLIGHT HENCE THE SECURE
JENKINS X CICD HAS BEEN OMITTED FOR NOW

!9