Kubernetes Security

Hunter Nield
Kubernetes Singapore Meetup - January 2019
Topics
● Containers
● Operating Systems
● Kubernetes
● Benefits
Containers
Containers
● Minimal - Typically running a single process. Less code, less chance of vulnerability
● Task Specific - Predefined operation (app, ports, volumes, etc) making anomaly detection easier
● Isolated - Utilizing the built in kernel resource isolation features (cgroups and namespaces)
● Reproducible - Understand how the container is built
Are VMs are more secure than
Containers?
Measuring Cloud Security
● James Bottomley from IBM blogged about the measurement of Container and VM Security
● Shows the impact of various actors in the Cloud Ecosystem
Attack Profiles
Vertical Attack Profiles (VAP)
● All code traversed to deliver a service
(frontend, DBs, Kernel, etc)
● Typically the domain of the Tenant in a Cloud
Service
● More code increases the risk of exploitable
vulnerabilities
Horizontal Attack Profiles (HAP)
● The infra provider code (Kernel, Hypervisor,
Container runtime)
● Typically the domain of the Cloud provider or
Ops team
● More code increases the risk of breakouts and
containment failures
● HAP = (Density of Kernel bugs) x (Syscalls)
Baremetal

● Tenant manages the VAP

● Minimal HAP since the hardware is
airgapped
Shared Environments

● Worst possible option

● Significantly higher VAP
○ Middleware and Libraries
● Significantly higher HAP
○ Single kernel with no isolation
VMs

● VAP higher than Baremetal for tenants

○ 2 x Kernels
○ Guest Virtual Hardware (harder to
detect by Cloud provider)
● HAP is higher due to Hypervisor
Containers

● VAP almost identical to Baremetal for

tenants.
● HAP is higher (10x more syscalls)
unless sandboxing is used (seccomp,
etc)
(Controversial)
Conclusion
● “Containers are by far the most secure
virtualization technology available today from
the Tenant perspective”

● HAP is higher for containers but responsibility

is with Infra provider to mitigate and patch

● Keep an eye on technologies like Nabla &

Gvisor (and now Amazon’s Firecracker)
Operating System
Operating System
● Minimal Container Operating Systems
○ RedHat CoreOS, Rancher
● Immutable OS
○ Read Only Filesystems
○ Replace OS for upgrades or errors
● Audit Docker or Container Runtime (Docker bench, etc)
● Disable SSH Access
Kubernetes
Threat Model
● External Attackers
● Application or container compromise (VAP and HAP)
● Compromised User or Credentials
Workload Isolation
Cluster

Node

Namespace

Pod

Container Container
Segregation
● Namespace-level
○ Visibility and Access via RBAC, Quotas and Secrets
● Node-level
○ Sensitive workloads via Affinity and Taints
● Pod-level
○ Limit communication via Network Policies and Service Mesh
Access Control
● Authentication
○ Humans - managed outside of Kubernetes and delegated to LDAP, OIDC, etc
○ Apps - managed inside of Kubernetes with Service Accounts
● Authorization
○ RBAC - Limit who can do what in the cluster
● Access
○ Cluster access through limited access points (API, Kubectl or UI) with TLS and audit trails
● Audit
○ Track all operations by Users and Service Accounts
○ Can be used to generate RBAC rules to limit access
Image Security
● Image Vulnerability Scanning is critical for images
○ CoreOS Clair / Twistlock / Aquasec can check for vulnerable packages
○ Some products also cover policy based scanning (NIST, HIPAA, etc)
● Scanning images
○ In registries
○ Stages in CI pipelines
○ Running containers
● Immutable images mean that a security update is just a rebuild away
● Kubernetes Admission Controllers can provide enforcement for vulnerable images
○ See Grafeas for an upcoming industry spec for artifact metadata
Pod Security
● Run Pods to avoid breakout...
○ with known origin and vulnerability scans
○ using least privilege to run the workload
○ mounting only minimal host volumes
● Implement PodSecurityPolicy and SecurityContext
○ Mandatory Access Control - Seccomp, SELinux, Apparmor
○ Disable root or privilege escalation
○ Read only filesystems
Network Security
● NetworkPolicy enables more granular firewalling between Apps and Services
○ More flexibility and control than traditional ‘tiering’ with firewalling
○ Moves control to the App Devs/Operators
○ Network Ops and Auditors can view YAML instead of invisible firewall rules
● Service Meshes take network security further
○ Workload identity - Traditional IPs are not a secure way to identity services in a network
○ Automatic TLS between services
○ Monitoring and Tracing - Get performance and anomaly metrics
○ Advanced Ingress and Egress filtering - eg. limit outgoing traffic to whitelisted secure URLs
● Automatic SSL certificates for internet facing services
○ Ingress controllers can generate and renew Let’s Encrypt or Vault certificates
Istio

Web App Query Backend

Egress
Label: V1 filtering
https://
Sidecar Sidecar Google.com

90% Traffic Pod (V1) Pod

SSL
Termination

Web App New DB

Istio 10% Traffic
Ingress
Sidecar Sidecar
mTLS
Pod (V2) Pod

User-agent:
Android
Mobile API Legacy DB

Sidecar Sidecar

Pod VM
Runtime Security
● Since containers are task specific it is much easier to detect anomalies
○ Shell spawned in container
○ File directories and paths that can be accessed and/or written.
○ Binaries that should be running.
○ External services that should be contacted and in general network connections.
● Tools like Falco can monitor the behaviour inside containers, log to SIEMs and raise alerts
○ Workflows can then be triggered to isolate nodes or relaunch containers
Secrets
● Kubernetes can store secrets and inject into Containers
○ ENV variables are not a secure method for injecting secrets into containers
○ Volume mounting is a preferable option (since it mounts in a tmpfs)
● Kubernetes by default doesn’t encrypt secrets in the data store
○ Recommended to set up encryption at rest or integrate advanced tooling like Vault
User Benefits
Cluster Operators
● Small attack surface and faster updates with Container OSes
● Standardised platform for running Applications and Services
● Audit and control over User access and Workloads
Developers
● Common delivery format with easy reproducibility and deployment
● Greater control over network and application security
● Vulnerability scanning during development and during run-time
● Real time visibility into intrusion or unusual behaviour in containers
Network Operators
● Visibility, audit and enforcement for network topology, services and segmentation
● Death to SDN (Spreadsheet Defined Networking)
● Automatic encryption and identity between Applications
● Detailed monitoring going beyond packet capture into protocols
Security & Audit Teams
● Visibility into the present state of environments through source controlled declarative configurations
● API driven for compliance automation
● Audit logging for any changes in the system
● Real time visibility into intrusion or unusual behaviour in containers
In Summary
A Glorious Secure Future with Kubernetes!
● Unicorns for everyone!
● Ice cream for breakfast
● Kubernetes comes with a free pony!
Not quite....
What’s missing...
● Collection of tools vs Product
○ Vanilla Kubernetes has the basics with additional tools needed
○ Openshift and other Kubernetes distros provide a limited but more complete solution
● Fast changing landscape
○ New tools daily
○ Difficult for commercial offerings to keep up
Discussion...