WHITE PAPER

Kubernetes:
The Good, The Bad
& The Misconfigured
How to eliminate Kubernetes misconfigurations to
reduce risk and improve reliability and cloud efficiency
 DATASHEET // DATADOG & FAIRWINDS
KUBERNETES: INSIGHTS:
THE GOOD, DATASHEET
THE BADA &POWERFUL
THE COMBINATION
MISCONFIGURED // 2

TABLE OF CONTENTS

Introduction.............................................................................. 3

Proper Configuration Counts.................................................... 4

Infrastructure as Code (IaC) Scanning Helps.............................. 5

IaC & Security.............................................................................. 5

IaC & Efficiency............................................................................ 6

IaC & Reliability............................................................................ 6

Misconfigurations Minimized.................................................... 8
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS:
THE BADA &POWERFUL COMBINATION
KUBERNETES: THE GOOD, THE MISCONFIGURED // 3

INTRODUCTION
Although security is often the most pressing topic in Kubernetes,
it is inextricably linked to issues of efficiency and reliability. To run
Kubernetes effectively and successfully, organizations must work
with intention, as workloads are not secure, reliable or efficient by
default. While Kubernetes provides some level of self-healing when
things go wrong, applications require the proper settings to work
correctly and to take full advantage of the built-in capabilities in
the underlying Kubernetes platform.

Security, efficiency and reliability are the three high-level

constructs inextricably linked to successful Kubernetes
deployments. When these three concerns are not properly
addressed through best practices, critical elements like cost
optimization, performance—and yes, reliability issues—are
severely impacted. These key areas must be addressed as one
interconnected challenge, to be solved through the singular
strategy of proper configuration. As a result, practitioners need to
perform a lot of different types of checks around general reliability
of workloads to ensure they are resource efficient and cost
effective. Therefore, assessing all dimensions of the configuration
posture is key to running successful clusters.
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS:
THE BADA&POWERFUL COMBINATION
KUBERNETES: THE GOOD, THE MISCONFIGURED // 4

PROPER CONFIGURATION COUNTS

Our recent Kubernetes Configuration Benchmark Report tells us that not all

35%
organizations have found steady footing with proper configuration of their
Kubernetes environment, nor are they even half way there. For example, only
35% of organizations have correctly configured most (>90%) of their workloads
of organizations have with liveness and readiness probes. These probes provide a way for Kubernetes
correctly configured to understand whether an application is alive and ready to serve traffic—and
most (>90%) of their take remedial actions if not. As a result, not having these probes in place can
lead to serious reliability problems.
workloads with liveness
and readiness probes. It is well known that Kubernetes can automatically scale resources up or down
in response to varying workload demand. Indeed, this is one of the primary
features of the platform. What is less well known is the fact that containers have
built-in configuration settings for determining the amount of CPU and memory

42%
resources they use (via resource requests and limits). These settings in essence
override some of the auto scaling capabilities of the underlying platform and can
therefore lead to under provisioning of the workloads. While under provisioning
of organizations today can cause performance issues, overprovisioning can lead to potentially dramatic
manage to lock down inefficiency and cost overruns.

most of their workloads. Furthermore, minor misconfigurations can produce major security holes if they
are not found and addressed. For example, a common vulnerability is containers
running with more security permissions than required, e.g., root-level access.
Under particular configurations, a container may escalate its own privileges.

54%
are leaving over half
Because configurations are not established by default, security-conscious teams
need to explicitly set them.

Only about 42% of organizations today manage to lock down most of their
workloads, while some 54% are leaving over half their workloads open to
their workloads open to
privilege escalation—and thus, security holes.1 Problems with configuration can
privilege escalation—
become increasingly painful over time, as they consume considerable resources
and thus, security holes. to fix. What, at first, feels like a few little issues to fix, quickly morphs into full-
blown Kubernetes chaos in the form of security vulnerabilities, wasted resources
and reliability concerns.

1. Benchmark Report 2021: Kubernetes Configuration

 DATASHEET // DATADOG & FAIRWINDS INSIGHTS:
THE BADA &POWERFUL COMBINATION
KUBERNETES: THE GOOD, THE MISCONFIGURED // 5

INFRASTRUCTURE AS CODE (IAC) SCANNING HELPS

While configuration validation, also known in the industry as infrastructure
Proper Kubernetes
as code (IaC) scanning, might be manually doable in a small team with one or
configuration is vital two Kubernetes clusters, the problem becomes increasingly challenging as
to the success of organizations scale with numerous development teams deploying to multiple
cloud native adoption. clusters. DevOps teams, along with platform and security leaders, can quickly
Without it, improving lose visibility and control into what is happening. This reality points to the need
application security, for automation and policies to enforce consistency and provide the appropriate
reliability and efficiency guardrails across the organization.

is basically impossible. The beauty of Kubernetes is its customization—but that customization can
cause risk, downtime or wasted resources. The bottom line is proper Kubernetes
configuration is vital to the success of cloud native adoption. Without it,
improving application security, reliability and efficiency is basically impossible.

HOW DOES IAC SCANNING AFFECT SECURITY?

It offers a tool to proactively identify security holes before they
become full-blown breaches.

The beauty of containers as a concept is they are a self-contained package

of everything the software needs to run in production. This feature greatly
facilitates the hand-off of software from development to operations and
speeds the delivery process. As organizations become increasingly familiar
with Kubernetes, it’s important to avoid letting security vulnerabilities (or other
problems) slip into production because of negligence or lack of experience. A
single workload may require significant configuration to ensure a more secure
and scalable application. Stack on technical debt and organizational hurdles, and
even the most experienced Kubernetes professionals struggle to get things right
every time.

Human error is the most cited cause of security breaches. When developer-
friendly (unsecure) default configurations are combined with human oversight,
container security lies in the balance. Moreover, configuration management
poses a unique challenge for Kubernetes users because it requires more
consideration. While many tools are available for vulnerability scanning of
container images, proper configuration and oversight demands careful handling.
Even though practitioners may understand the need to avoid deploying the
Kubernetes dashboard, configuring a pod’s security content, or implementing
RBAC are other examples of the challenging setting these teams are facing.
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS:
THE BADA&POWERFUL COMBINATION
KUBERNETES: THE GOOD, THE MISCONFIGURED // 6

Although the world of cloud native technologies and Kubernetes is still relatively
new, the core business challenge remains the same. Organizations must figure
out how to accelerate development speed while also maintaining robust security
practices. These two business objectives are still vying for equal attention in the
container space.

HOW DOES IAC SCANNING AFFECT EFFICIENCY?

It provides the necessary visibility to proactively identify the
inefficiencies wasting money and resources.

Misconfigurations of Kubernetes workloads often involve inefficient provisioning

of compute resources—and that leads to an oversized bill for cloud compute. To
maximize CPU efficiency and memory utilization for a workload, teams need to
set resource limits and requests properly, as mentioned earlier. But here is the
catch—knowing the right limits to set for smooth application performance can
be tricky at best. This is where visibility comes in.

Gaining visibility into application resource usage can help teams better
understand how their application performs with different CPU and memory
settings. These can then be adjusted to improve app performance or to increase
the efficiency of Kubernetes compute resources, ultimately helping organizations
save money in the cloud and capacity in their data centers.

HOW DOES IAC SCANNING AFFECT RELIABILITY?

It gives DevOps teams methods for avoiding downtime and
production incidents.

In Kubernetes, reliability is about building a stable platform so development

teams can streamline their development process and ship applications faster.
Platform engineering and operations teams are responsible for monitoring the
health of Kubernetes clusters, which is implemented and orchestrated through a
set of best practices. Platform engineering teams must partner with development
to ensure workloads are configured reliably from the start—and truth be told,
Kubernetes misconfigurations happen a lot.
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS:
THE BADA &POWERFUL COMBINATION
KUBERNETES: THE GOOD, THE MISCONFIGURED // 7

Kubernetes offers a framework where distributed systems are built with

microservices and containers to run applications reliently. This model means
separate teams own different layers of the stack, a fundamental concept of
Kubernetes service ownership, and developers are specifically responsible
for getting their applications to Kubernetes with proper configurations. This
pervasive DevSecOps-like model of service ownership frees up operations teams
from handling deployment configuration and allows them to focus on policy
enforcement and actionable developer feedback.

Workload configuration, typically made in YAML files and Helm charts, affect
the security and reliability of services, as well as the efficiency of workloads in
a cluster. There are numerous factors to consider when assembling a stable and
reliable Kubernetes cluster, including the potential need for application changes
and alterations to cluster configuration. These considerations include things like
setting resource requests and limits, autoscaling pods with the right metrics and
using liveness and readiness probes.

Infrastructure as code scanning solutions, such as those available in Fairwinds

Insights, can inspect YAML and Helm configurations when developers make a
pull request. Like traditional infrastructure as code scanning solutions, Insights
examines configuration for security violations, like privilege escalation. The
software goes further by also incorporating efficiency and reliability checks for
platform engineering teams, who rely on them for running stable and scalable
infrastructure.

Here are some examples of reliability checks that

enable correct configuration:
• Limits must be set on resource consumption to keep pods from
consuming all the available memory and CPU on a node, otherwise
known as the “noisy neighbor problem.”
• Containers across multiple nodes and availability zones should be
scheduled in the cloud for high availability.
• Anti-affinity should be used to constrain which nodes are eligible
for scheduling based on pod labels, rather than on nodes.
• Fault tolerance must be planned by deploying redundant instances
to avoid a single point of failure.
• Liveness and readiness probes should be applied to ensure the
availability of services and to check cluster performance.
 DATASHEET // DATADOG & FAIRWINDS INSIGHTS:
THE BADA &POWERFUL COMBINATION
KUBERNETES: THE GOOD, THE MISCONFIGURED // 8

MISCONFIGURATIONS MINIMIZED
The answer to how misconfigurations can be minimized is multi-faceted one.
Large organizations will find it is nearly impossible to manually check each
security configuration and assess its risk. Because Kubernetes defaults tend to
be inherently open and unsecure, it is important to avoid using these default
settings until all security implications—and how they impact the overall risk
tolerance—are clearly understood.

Helpful guidance and a useful framework for hardening an environment can be

found in various objective, consensus-driven security guidelines for Kubernetes
software, such as the CIS Benchmark. When these best practices are paired
with risk-based policies integrated into the CI/CD pipeline, container security
improves. Commits or builds that do not meet minimum security requirements,
or provide guardrails for developers, can be halted.

Protecting Kubernetes clusters and workloads at runtime, to ensure security,

efficiency and reliability, demands a multi-pronged approach using defense-in-
depth. Part of this solution comes from finding a SaaS orchestration platform
with the ability to establish effective governance, streamline development and
operations and provide a better (and safer) user experience.

Because misconfigurations are so common, building a stable, reliable and secure

cluster only happens when the best practices outlined here are followed. And
this level of governance only comes through a trusted partner, well-versed in the
process of unifying teams, simplifying complexity and building on Kubernetes
expertise to save time, reduce risk and configure with confidence.

Fairwinds Insights offer this level of professional expertise and partnership. As

a security and governance platform for Kubernetes, Insights provides DevOps
teams with a safety net for scalability, reliability, resource efficiency and security
while also empowering developers to innovate and ship faster. DevOps teams
can then prevent misconfigurations throughout the CI/CD pipeline and provide
remediation advice to developers, free from manual intervention. With Fairwinds
Insights, managing multiple clusters and teams across the enterprise becomes
easier—and in many cases, possible—as it operationalizes open source tools
into a single platform for better oversight and management.
WHY FAIRWINDS
Fairwinds is your trusted partner for Kubernetes security, policy and governance. With Fairwinds, customers
ship cloud-native applications faster, more cost effectively, and with less risk. We provide a unified view
between dev, sec, and ops, removing friction between those teams with software that simplifies complexity.
Fairwinds Insights is built on Kubernetes expertise and integrates our leading open source tools to help you
save time, reduce risk, and deploy with confidence.

WWW.FAIRWINDS.COM