Container Security:

Going Beyond
Image Scanning
In theory at least, containerized applications
are more secure than previous generations of
applications. It’s a lot easier to rip and replace
containers that have encapsulated vulnerable
code, behave in a suspicious or malicious way
indicative of an active breach, or contain high-
risk misconfigurations, than it is to patch an entire
monolithic application. Containers are designed to
make it possible to build and deploy applications
and incremental updates more easily using
microservices that isolate specific functionality in
a modular fashion. In short, microservices make it
easier to rapidly discard a container and replace it
with a more secure version.

However, an application environment can have

thousands of containers — built from base images
that may or may not have originated from trusted
sources — and each container image is composed
of layers of files. Finding the images that have
vulnerabilities requires a significant amount of time,
effort and expertise, but time spent scanning images
at the build stage paves the way for a more secure
application. For this reason, one of the first steps
most organizations take to secure their containerized
applications is to scan their container images for
known vulnerabilities before pushing it into their
registry.

Image scanning is one piece of a much larger

security puzzle that must be solved in order to
ensure your application is protected throughout the
container life cycle.

2
A recent survey of over 400 IT professionals who have Most common
experience working with containers and Kubernetes clusters Kubernetes container

67+33+S
finds a full 90% have dealt with some type of cybersecurity cyberscurity issues
incident involving containers. The most common issues
involved misconfigurations (67%), followed by an actual


major vulnerability (22%), runtime threats (17%) and failed
audit (16%).

Naturally, security will be a concern whenever an 67%

22+78+S
organization adopts an emerging technology. A third of the
Misconfigurations
survey respondents (34%) said they have security concerns
related to their organization’s container strategy. Those
concerns are already having a major impact—nearly half of

 22
respondents (44%) said they have had to delay deploying an
application due to security concerns. 22%
%
Organizations of all sizes are trying to address those

17+83+S
concerns proactively by shifting responsibility for Major vulnerabilities
application security toward developers, otherwise known as
DevSecOps, and implementing security controls spanning
the entire container life cycle, from build, to deploy, to
17%
17%
runtime.

In this e-book, we discuss security best practices that

you should follow while building images and scanning for


16+84+S
vulnerabilities. We will then expand our discussion beyond Runtime incidents
image scanning and identify opportunities for you to
protect your containerized applications and infrastructure
throughout the software life cycle. 16%
16%
Note: this e-book assumes Kubernetes to be the container
orchestrator but many of the recommendations provided are
applicable to most container environments regardless of the

management tool being used. Failed audits

3
SECURING CONTAINER IMAGES
Beyond providing developers with tools to scan images for vulnerabilities, IT organizations also
need to secure the infrastructure entrusted to run those container images. As cybercriminals
become more familiar with container technologies, they are discovering that the most efficient
way to compromise containers is to attack the infrastructure on which they are deployed.
Cybercriminals are known to employ misconfigured Docker application programming interfaces
(APIs) to compromise entire hosts. Cybercriminals can then spin up whatever Docker container
images they like.

BEST PRACTICES FOR
SECURING CONTAINER IMAGES
Fortunately, a set of best
practices for securing container
environments is now emerging.
The best place to secure container
images will always be at the
point where they are created,
which is why static and dynamic
image scanners are now being
integrated with continuous
integration/continuous delivery
(CI/CD) platforms. The goal is to
make security checks a natural
extension of the application
development process. Instead of
throwing code “over the wall” for
cybersecurity teams to review,
application developers are now
scanning for vulnerabilities as they
write code and as that code is
committed to a CI/CD platform.
Once that code is deployed in a
runtime environment, it should be
subjected to additional periodic
scans to ensure the environment
remains secure.
The first thing
developers should be
made aware of is the
need to use secure
base images whenever possible. A
secure base image provides some
peace of mind because they are
automatically patched by the entity
that created them. Most developers
will employ a secure base image as
the foundation for an application by
layering additional images on top of
the base image. If the base image is
secure, it can make securing the rest
of the application that much easier.
IT teams should also
endeavor to keep
the base images
they employ to a minimal size
by removing non-essential
components. Package managers,
networking tools, shells, debuggers
and other application development
tools should never be included
in a production environment. The
smaller the base image, the less
attack surface there is to defend.
Smaller images are also less
likely to be impacted by zero-day
vulnerabilities if and when they arise.
Restricting the image to required
binaries, libraries and configuration
files is always a good idea.
Development teams
should also take care
to never bake in any
secrets to images.
Secrets include TLS certificate
keys, cloud provider credentials,
SSH private keys and database
passwords. Anyone who can pull the
image can extract the secret. Making
sensitive data available only at
runtime enables developers to reuse
the same image in different runtime
environments, while still being able
to update expired or revoked secrets
without having to reconstruct the
image. Most organizations manage
secrets via a pod in their Kubernetes
cluster or a dedicated secret
management system.

4
Establishing a comprehensive
DevSecOps workflow for container
application is essential. The
challenge is not only making
sure those workflows are applied
consistently, but also educating
developers on why a build failed
because of an unaddressed
security issue.
CONTAINER REPOSITORIES
MATTER
Developers pull
container images
from both public and
private repositories,
also known as registries, all the
time. The most widely employed
public repository is Docker Hub.
However, only a small percentage
of the container images available
on any public repository have been
validated; therefore, organizations
need to be aggressive in inspecting
container images pulled from
any public repository. It’s not
uncommon for container images
on a public repository to have at
least one or more known major
vulnerabilities, often because they
were constructed with outdated
software modules.
IT teams would be well-
advised to standardize
on a single container
registry through
which they can apply security and
compliance controls consistently.
That approach makes it possible to
ensure whatever container images
that are reused have been scanned
for vulnerabilities and are compliant
with all application regulatory
requirements.
Each time a container
image is updated
it should also be
automatically
scanned for known
vulnerabilities. At a minimum, each
container image should be scanned
at least every 90 days. Organizations
will have to decide for themselves
how much risk is associated with
each vulnerability discovered.
DevSecOps teams should have
access to information describing the
severity of the vulnerability as well
as contextual information describing
how the vulnerabilities may impact
their production environment,
including the different versions and
distributions of Kubernetes clusters.
The latest version of Kubernetes
is 1.19, and some organizations
have clusters running versions of
Kubernetes as old or older than
1.11. There are also more than 100
certified distributions of Kubernetes,
each of which have their own
unique configurations.
Vulnerability assessment tools
need to be able to pinpoint where a
vulnerability lies along with specific
recommendations for remediation.
5
HARDENING THE DEPLOYMENT
Once you’ve built your images, the next step is to deploy them. To achieve security in the
deployment phase, you need to understand and manage risk factors including those posed by
the contents of the images being deployed and the configuration of individual containerized
workloads.

While default orchestrator and container engine settings provide a reasonable level of
security in some areas, other settings require specific configurations to lock them down. The
right container security solution should not only display violations of standard policies, but
also provide the DevOps team with recommended changes to quickly correct unnecessary
risks or misconfigurations.

Only with all of this data can

Individual workloads have rich configuration options you get a full understanding
that affect the security posture of the application. To of your risk posture, so
fully understand the risk posed by an application, you you know where to target
need to know: remediation and hardening
efforts. And only with
• What it is (including information about the image, checks and enforcement on
all of these attributes can
such as components or vulnerabilities).
you ensure that workloads
are deployed securely.
• Where it came from (including the registry and tag).

• How it’s deployed (including user identity, privilege
level, and other configurations).
• What it can access (including secrets, volumes, and
other infrastructure components such as the host or
orchestrator API).
• Whether it complies with your policies and security
requirements.
Within a cluster, sensitive
workloads should also be
separated logically and
physically. Different groups of
applications, and services with
different security needs, should be
deployed in separate namespaces
as a first level of isolation. The most
sensitive workloads should run
on dedicated nodes so that less
trusted workloads are not able to
affect their operation.

6
BEST PRACTICES FOR HARDENING
THE DEPLOYMENT
USE NAMESPACES TO PROVIDE
WORKLOAD ISOLATION
Namespaces are a key boundary for network
policies, orchestrator access control restrictions,
and other important security controls. Separating
workloads into namespaces can help contain
attacks and limit the impact of mistakes or
destructive actions by authorized users.
UNDERSTAND HOW PERSISTENT
STORAGE IS CONFIGURED AND
USED
Persistent storage is a key persistence vector in
otherwise ephemeral container environments.
Persistent storage may also store valuable, sensitive
data. For these reasons, you should be able to
identify the use and configuration of persistent
storage to enhance your security posture.
UNDERSTAND HOW HOST
MOUNTS ARE USED
Host mounts are another vector that adversaries
use to establish persistence or interfere with
other containers. Host mounts are sometimes
unintentionally writable or overbroad. You should
be familiar with the use and configuration of host
mounts.
IDENTIFY EXTERNAL NETWORK
EXPOSURES
External network exposure opens services to a
broader pool of potential adversaries. Removing
unnecessary exposures can prevent exploits,
especially when new vulnerabilities emerge.
UNDERSTAND YOUR NETWORK
SEGMENTATION RULES
Orchestrators typically provide relatively open
networking between components — for instance,
Kubernetes by default allows every pod to contact
every other pod. Network segmentation policies
are a key security control that can limit the ability
of an attacker to move laterally through a container
environment.
UNDERSTAND HOW SECRETS
ARE BEING USED
The container infrastructure can enforce access
controls to prevent secrets from being read by
unrelated deployments, but that step helps only if
deployments mount just the secrets they actually
need. Visibility into secrets helps you find this type
of unnecessary exposure and can expose other
weaknesses, such as short key lengths.
7
BEST PRACTICES FOR HARDENING
THE DEPLOYMENT
ASSESS PRIVILEGES USED
BY CONTAINERS
The capabilities, user identity, and privileges
granted to container processes can allow or block
many security attack vectors, and you should
examine these settings to minimize risk.
ASSESS IMAGE PROVENANCE,
INCLUDING REGISTRIES AND
OTHER ATTRIBUTES
It’s highly recommended for your teams to know
where the code in your environment was deployed
from and implement controls that prevent
deployment from untrusted sources or registries.

ENSURE OLD (AND POTENTIALLY ASSESS IMAGE SCAN STATUS

VULNERABLE) IMAGES ARE
NOT BEING USED
Your security assessment should include the age
of images in their risk profiling, since older images
can contain more vulnerabilities than fresher ones.
You should set up alerts for when very old images
are deployed.
BEFORE, DURING, AND AFTER
DEPLOYMENT
It’s important to enforce distinct checks based
on last vulnerability scan date because images
that haven’t been scanned recently may contain
vulnerabilities that are discoverable but not yet
reported for the image.

ASSESS OPERATIONAL BEST

ASSESS RESOURCE REQUESTS
AND LIMITS FOR DEPLOYMENTS
Deployments without resource limits can cause
availability issues for themselves or for neighboring
applications, especially if they are compromised
and used to run cryptocurrency miners or similar
payloads.
PRACTICES, SUCH AS LABELS
AND ANNOTATIONS OR THE USE
OF THE “LATEST” TAG
Workloads that don’t specify standard annotations
or labels are harder to understand, making it more
difficult to remediate security issues. Using tags
like “latest” increases the risk of new code being
accidentally deployed.

8
BEST PRACTICES FOR HARDENING
THE DEPLOYMENT
IMPLEMENT CHECKS TO DETECT AUTOMATICALLY NOTIFY APPROPRIATE
COMMON CONFIGURATION TEAMS BASE ON DEPLOYMENT
PROBLEMS METADATA
CHECK-SQUARE High-severity vulnerabilities. Many organizations annotate deployments with
CHECK-SQUARE Image scan age and other best practices. the name, email alias, or Slack channel of the team
responsible for an application. You should be able
CHECK-SQUARE Abnormal privilege levels. to use this metadata effectively to prevent security
CHECK-SQUARE Security-relevant image components, teams from having to manually triage and forward
including package managers and download alerts and findings.
tools.
CHECK-SQUARE Key vectors for adversary persistence attempts,
such as sensitive host directory mounts.
CHECK-SQUARE Common bad practices, such as distributing BLOCK DEPLOYMENTS THAT FAIL
secrets in environment variables. CRITICAL CHECKS
Configuration options evolve over time and The best way to ensure ongoing compliance with
may require significant research to develop policies is to prevent noncompliant workloads from
appropriately specific and understandable being deployed.
policies. Look for out-of-the-box policies in your
security platform to help your team initiate its
security efforts. A solid base of pre-built policies
allows your team to instead focus on building
custom policies in areas they know best: your
applications and practices.

9
SECURING RUNNING CONTAINERS
The goal of the security controls instituted earlier in the life cycles, and described above, is to
minimize risk and prevent attacks, but a successful container security program must also detect
and respond to issues as containers run.

In runtime, security efforts should focus on protecting running deployments by providing

both visibility and detection features. You must monitor the most security-relevant container
activities, including:

Process activity Network communications Network communications

among containerized between containerized
services services and external
clients and servers

Container image and application deployment formats allow developers to better specify
their intent and enable easier introspection into what’s deployed. Effective container security
solutions must take into account the context accumulated during the Build and Deploy
phases of the container life cycle to deliver more effective and accurate runtime detection.

In addition, effective runtime security requires behavioral analysis of container runtime

activity using sophisticated techniques that establish risk, detect and respond to active
attacks, and track attack progression, in addition to basic approaches that include
generating allowlists for process executions, blocking and static log analysis.

10
BEST PRACTICES FOR PROTECTING
RUNTIME CONTAINERS
MONITOR RUNNING GET VISIBILITY INTO ACTIVE
DEPLOYMENTS FOR NEWLY NETWORK TRAFFIC BETWEEN
DISCOVERED VULNERABILITIES ORCHESTRATED MICROSERVICES
Vulnerability data on running — PREFERABLY ORGANIZED BY
containers must be continually CLUSTER, NAMESPACE, AND
updated rather than relying on a single image DEPLOYMENT
vulnerability scan at a particular time. New Microservices and containerized applications
vulnerabilities may be disclosed publicly after typically make extensive use of cluster networking
deployment, so images need to be periodically to cooperate with other services. Active network
rescanned, and any new results need to be included traffic is a key way to understand how applications
in the risk assessment for the deployment. are interacting and spot unexpected contact.

COMPARE AND ANALYZE

UNDERSTAND ALL PROCESSES DIFFERENT RUNTIME ACTIVITY
EXECUTED IN CONTAINERIZED IN PODS OF THE SAME
DEPLOYMENTS DEPLOYMENTS

Processes are an important indicator of security Microservices and containerized applications

and operations-relevant container activity. In typically make extensive use of cluster networking
containers, ad-hoc process launches are less to cooperate with other services. Active network
frequent and more suspicious than they would be traffic is a key way to understand how applications
in other types of infrastructure. are interacting and spot unexpected contact.

ENFORCE CUSTOM POLICIES THAT COMBINE RUNTIME CRITERIA

WITH ATTRIBUTES FROM BUILD- AND DEPLOY-TIME
Build- and deploy-time context, such as orchestrator annotations or image details from
scanners and registries, can help security teams build more effective and targeted
policies. For example, images from public registries might be less trusted, and interactive
debugging activities might be more restricted in sensitive applications.

11
BEST PRACTICES FOR PROTECTING
RUNTIME CONTAINERS
IMPLEMENT RUNTIME DETECTION
POLICIES FOR ATTEMPTED OR
SUCCESSFUL:
CHECK-SQUARE Privilege escalation.
CHECK-SQUARE Network reconnaissance.
CHECK-SQUARE Package installation.
CHECK-SQUARE Cryptocurrency mininig.
CHECK-SQUARE Modification of host configuration.
CHECK-SQUARE Execution of reverse shells or other remote
access tools.
CHECK-SQUARE Data exfiltration.
DETECT UNEXPECTED ACTIVITIES
THAT INDICATE AN ADVERSARY
MAY BE GAINING A FOOTHOLD,
ESTABLISHING PERSISTENCE,
ESCALATING PRIVILEGE, MOVING
LATERALLY, OR ACHIEVING THEIR OBJECTIVES
Adversary behaviors may not match a specific
denylist item or known vector, but should still be
identified using more sophisticated techniques that
compare activity to each application’s baseline.
AUTOMATICALLY ALERT EXTERNAL
SYSTEMS (E.G., EMAIL, PAGERDUTY,
SLACK, GOOGLE CLOUD SCC AND
SIEM SYSTEMS) WHEN A POLICY IS
VIOLATED
You should immediately alert the right responders
once an attack is detected and the recommended
response should be based on the severity of the
policy violation.
NEVER PATCH, AND INSTEAD USE
KUBERNETES-NATIVE CONTROLS
TO REPLACE SUSPICIOUS
INSTANCES OF RUNNING
APPLICATIONS.
Don’t let anyone patch/update a running container;
instead, it’s faster, easier and more secure to “scale
to zero” and then restart. In a runtime environment,
just kill the pod and relaunch a new one. Containers
are meant to be immutable, so when suspicious
activity is detected, instruct Kubernetes to scale
suspicious pods to zero, then restart instances of
problematic applications.
12
THE KUBERNETES INFRASTRUCTURE
To understand how to effectively secure your Kubernetes infrastructure, it is informative to
understand the architecture of Kubernetes itself as well as where and how to focus efforts on
valuable mitigations, especially those which require administrator or user configuration when
provisioning clusters. Kubernetes is a robust yet complex infrastructure system for container
orchestration, with multiple components that must be adequately protected.

Each Kubernetes cluster consists of two sets of components: (1) the control plane which
is used to manage operations throughout the cluster, and (2) the cluster’s worker nodes
which run containerized applications in pods. Also, to achieve high availability and resiliency,
especially for production cluster environments, both sets of components can be deployed
across multiple machines that form a cluster.

KUBERNETES CONTROL PLANE

CLOUD
KUBE- CLOUD-
CONTROLLER CONTROLLER
MANAGER MANAGER

KUBE-API-SERVER KUBERNETES NODES

KUBELET KUBELET KUBELET

ETCD KUBE-
KUBE-PROXY KUBE-PROXY KUBE-PROXY
SCHEDULER

CONTAINER CONTAINER CONTAINER

RUNTME RUNTME RUNTME

13
SECURING KUBERNETES
CONTROL PLANE COMPONENTS
Considering that the Kubernetes control plane is designed to make global decisions regarding
a cluster’s operations, compromise of control plane components could easily result in complete
compromise of a cluster. Therefore, a critical starting point for any effort to secure Kubernetes is
to initially focus on protecting its control plane services.

The Kubernetes control plane includes the following components listed below, along with
recommended steps you can take to increase security for each component.

KUBERNETES API SERVER ETCD

(KUBE-APISERVER)
This is the central control plane component
that exposes the Kubernetes API and
receives and validates API requests for
resource creation and modification. It serves as the main
interface for cluster communications, since it is the only
cluster component that all control plane and worker
components can communicate with directly.
A number of vulnerabilities impacting the Kubernetes API
server have been and continue to be disclosed. In addition
to taking steps to mitigate these vulnerabilities by upgrading
Kubernetes versions or applying security patches, restricting
access to the Kubernetes API is key to securing clusters.
This can be done by:
• Setting up authentication for all Kubernetes API clients.
• Configuring authorization that determines permissions
regarding who can do what, using Kubernetes Role-
based Access Control (RBAC).
• Ensuring all API traffic is TLS-encrypted.
This control plane
component is the key value
store that Kubernetes uses
to store all cluster data,
including API objects and configurations.
Data stored in etcd needs to be adequately
protected using encryption and access
control since gaining visibility into this data
can reveal significant details about your
environment, including your containerized
applications.
Kubernetes supports encryption at rest of
sensitive data such as secrets, but only if it is
enabled. Additionally, strong authentication
and access controls must be enabled to
limit both read and write access to the data
store — while write access to etcd grants full
privileges on the entire cluster.

Audit logging in Kubernetes should also be enabled to

ensure that API events are recorded for further analysis or
investigation, as warranted.

14
 KUBE-SCHEDULER
This is the control plane component that handles scheduling of pods on worker nodes based
on resource availability; requirements, and allocation; affinity/anti-affinity, taints/tolerations
requests, and other policy constraints.

File permissions on the kube-scheduler pod specification and kubeconfig files should be restricted, and the
service should be configured to only serve HTTPS. Additionally, the service should be bound to a localhost
interface since it exposes health and metrics information which, by default, do not require authentication or
encryption. Communication with the API server should be done using the secured port, in addition to requiring
controls for authentication and authorization.

KUBE-CONTROLLER-MANAGER CLOUD-CONTROLLER-MANAGER
This control plane component runs This control plane component runs
processes that regulate, maintain processes that manage cluster
and adjust the state of things such dependencies in cloud provider
as nodes, pod replicas, services, environments for functions such as
and services accounts and access node termination, route setup and
tokens. load balancing.

File permissions on the kube-controller-manager pod File permissions on the kube-controller-manager

specification and kubeconfig files should be restricted,
and the service should be configured to only serve
HTTPS. It should be bound to a localhost interface
as well. You should also ensure that an individual
service account credential is configured per controller
in conjunction with Kubernetes RBAC to ensure the
control loops run with minimum required permissions.
Communication with the API server should be done
using the secured port, in addition to requiring controls
for authentication and authorization.
pod specification and kubeconfig files should be
restricted, and the service should be configured to
only serve HTTPS. You should also ensure that an
individual service account credential is configured
per controller in conjunction with Kubernetes RBAC
to ensure control loops run with minimum required
permissions. Communication with the API server
should be implemented using the secured port, in
addition to requiring controls for authentication and
authorization.

15
SECURING KUBERNETES
NODE COMPONENTS
Each Kubernetes node runs the following components:

KUBELET
This node component is the
primary node agent that manages
individual containers running in
pods and their state based on
PodSpecs passed via the Kubernetes API server, or
as a file with the kubectl command line tool.
KUBE-PROXY
This node component is the primary
node agent that manages individual
containers running in pods and their
state based on PodSpecs passed
via the Kubernetes API server, or as
a file with the kubectl command line tool.

Vulnerabilities related to the kubelet have been
and continue to be disclosed — these should
be addressed by upgrading kubelet versions,
applying available patches, and/or taking steps to
mitigate the particular threat vector.
Vulnerabilities related to the kubelet have been
and continue to be disclosed — these should
be addressed by upgrading kubelet versions,
applying available patches, and/or taking steps to
mitigate the particular threat vector.

Limiting access to the kubelet with strong
authentication and authorization is necessary
because its HTTPS endpoints expose APIs
that grant certain privileges on the node and
containers. By default, access is unauthenticated.
Limiting access to the kubelet with strong
authentication and authorization is necessary
because its HTTPS endpoints expose APIs
that grant certain privileges on the node and
containers. By default, access is unauthenticated.

CONTAINER RUNTME
This component enables the functionality required to start, run and manage containers on a
given node, such as communicating with the kernel, setting up cgroups, etc. Some examples of
runtimes supported by Kubernetes include: Docker, CRI-O and containerd.

The container runtime is what executes containers and, therefore, appropriate steps and security best practices
should be taken to harden it. See here for recommendations on securing Docker containers

16
SUMMARY
A foundational aspect of securing
your containers is to focus on the
earliest security gaps that can
arise at the build stage. And while
vulnerability scanning is one of
the most critical early steps you
should take to secure containers,
the security mission doesn’t stop
there. Securing containerized
applications from the time they’re
built until they’re deployed and
running as Kubernetes applications
requires a comprehensive security
program covering the entire software
life cycle and the underlying
infrastructure, too. Your security
tooling and workflows must be
adapted to fit how DevOps operate
so that security doesn’t become a
business inhibitor.

17
 Ready to see StackRox in action?
Get a personalized demo tailored for your business,
environment, and needs.

 http://www.devops.com
 https://twitter.com/devopsdotcom
 https://www.facebook.com/devopscom