258 BROUGHT TO YOU IN PARTNERSHIP WITH

Docker Security CONTENTS

•  Top Three OWASP Rules for

Container Security

Core Practices to Securing Containers •  CI/CD and Pre-Deployment Security

•   Runtime Container Security

•   Incident Response

•   Conclusion

BORIS ZAIKIN
SOFTWARE AND CLOUD ARCHITECT AT NORDCLOUD GMBH

Are containers insecure? Not at all. Features like process isolation with
user namespaces, resource encapsulation with cgroups, immutable
images, and shipping minimal software and dependencies reduce the
attack vector by providing a great deal of protection.
In this Refcard, we will introduce three OWASP rules that should
be implemented in projects that operate via Docker containers,
specifically:
1. Add –no-new-privileges flag

Container security tools are becoming hot topics in the modern IT space 2. Limit capabilities (Grant only specific capabilities needed by a
as early adoption fever is evolving into a mature ecosystem. Security is container)
an unavoidable subject to address when planning to change how we
3. Keep Host and Docker up to date
architect our infrastructure.
CI/CD AND PRE-DEPLOYMENT
This Refcard will lay out the basics of container security, assess key
SECURITY
challenges, provide hands-on experience with basic security options,
Pre-deployment security in CI/CD is quite essential. You can detect
and introduce some more advanced workflows.
vulnerabilities before they end up in your application with the following:

We’ll split container security into three sections, covering what to do at

CONTAINER IMAGE AUTHENTICITY
each step of your container security lifecycle:
There are plenty of Docker images and repositories on the Internet for
•  CI/CD and pre-deployment security every type of application under the sun, but if you are pulling images
•  Runtime security

•  Incident response and forensics

Lastly, we will take a look at Docker and Kubernetes security principles,

how cloud providers work with Docker containers, and what security
options are available.

TOP THREE OWASP RULES FOR

CONTAINER SECURITY
Before we jump into the rules, let’s further describe OWASP. OWASP
(Open Web Application Security Project) is a non-profit community
of security specialists. OWASP provides free security guidelines and
recommendations.

REFCARD | FEBRUARY 2022 1

Keep pace with the evolution
of cloud native architecture
Empower your organization with
a modern approach to cloud security
Quickly assess and manage the impact of vulnerabilities
like Log4j across multi-cloud environments

Leverage the combined power of agent-based

and agentless security
Get comprehensive cloud workload protection with a single solution.

Discover and manage risks faster

Identify and remediate misconfigurations across the application lifecycle.

Enable comprehensive coverage

for OWASP Top 10
Protect web applications and APIs against today’s
most pressing cloud native application security concerns.

FREE TRIAL
 REFCARD | DOCKER SECURIT Y

without using any trust and authenticity mechanism, you are basically •   Enforce mandatory signature verification for any image that is
running arbitrary software on your systems. going to be pulled or run on your systems.

•   You should disable image privilege escalation. You can do so by

You can use the following rules to prevent disasters:
adding the flag --security-opt=no-new-privileges when
•   Where did the image come from?
running your images.
•   Do you trust the image creator? Which security policies are they
If you run your Docker container in Kubernetes, you can add the
using?
allowPrivilegeEscalation flag:
•   Do you have objective cryptographic proof that the author is
actually that person? apiVersion: v1
kind: Pod
•   How do you know the image was not tampered with after you
metadata:
pulled it? name: backend-identity-service
spec:
Docker will let you pull and run anything you throw at it by default,
containers:
so encapsulation won't save you from this. Even if you only consume
- name: backend-identity-service
your own custom images, you want to make sure nobody inside the image: path/to/docker/image/backend-identity-
organization is able to tamper with an image. The solution usually boils service:1.0.3
down to the classical PKI-based chain of trust. securityContext:
allowPrivilegeEscalation: false
However, Docker and Docker Hub have integrated image vulnerability
scanning. This feature is based on the Snyc tool and scans your images Grant only specific capabilities needed for your containers. You can
after pushing it into the hub. Below, you can see the image of the use the --cap-drop flag. Also, you should not run containers with
vulnerability scan report: --privileged flag! The best and recommended approach would be
to drop all capabilities and then add one-by-one:

docker run --cap-drop all --cap-add CHOWN backend-

svc:1.0.3

Example: Deploying a full-blown trust server is beyond the scope of

this Refcard, but you can start signing your images right away.

1. Get a Docker Hub account if you don’t have one already.

2. Create a directory containing the following trivial Dockerfile:

# cat Dockerfile
FROM alpine:latest
Also, you can achieve the same result using the docker scan command
to scan your local images. You can run the following example and see Build the image:
the standard report:
# docker build -t /alpineunsigned
docker scan backend-app:1.1.3

Log into your Docker Hub account and submit the image:
However, if you need an extended report, you can run the following
# docker login
example:
[…]

docker scan --file Dockerfile backend-app:1.1.3 # docker push /alpineunsigned:latest

CORE PR ACTICES: Enable Docker trust enforcement:

•   The regular Internet common sense: Do not run unverified

# cat Dockerfile
software from sources that you don’t explicitly trust. FROM alpine:latest

•   Deploy a container-centric trust server using some of the Docker # export DOCKER_CONTENT_TRUST=1

registry servers.

REFCARD | FEBRUARY 2022 3 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | DOCKER SECURIT Y

Now, try to retrieve the image you just uploaded: images: “Actually, 7% of the images contained at least one
secret.”
# cat Dockerfile
FROM alpine:latest •   Deploy a Docker credentials management software if your
deployments get complex enough. Do not attempt to create
Using default tag: latest your own "secrets storage" (curl-ing from a secrets server,
Error: remote trust data does not exist for docker. mounting volumes, etc.) unless you really know what you are
io/<youruser>/alpineunsigned:
doing.
notary.docker.io does not have trust data for
docker.io/<youruser>/alpineunsigned Examples: First, let’s see how to capture an environment variable:

You should receive the following error message: # docker build --disable-content-trust=false -t /
alpinesigned:latest .
Using default tag: latest # cat Dockerfile
Error: remote trust data does not exist for docker. FROM alpine:latest
io/<youruser>/alpineunsigned:
notary.docker.io does not have trust data for
# docker run -it -e password='S3cr3tp4ssw0rd' alpine
docker.io/<youruser>/alpineunsigned
sh
/ # env | grep pass
Now that DOCKER_CONTENT_TRUST is enabled, you can build the
password=S3cr3tp4ssw0rd
container again and it will be signed by default:

# docker build --disable-content-trust=false -t / It's that simple, even if you are a regular user:
alpinesigned:latest .
/ # su user
# cat Dockerfile
/ $ env | grep pass
FROM alpine:latest
password=S3cr3tp4ssw0rd

Now, you should be able to push and pull the signed container without
Nowadays, container orchestration systems offer some basic secrets
any security warning. The first time you push a trusted image, Docker
management. For example, Kubernetes has the secrets resource.
will create a root key for you. You will also need a repository key for the
Docker Swarm also has its own secrets feature, which will be quickly
image. Both will prompt you for a user-defined password.
demonstrated below:

Your private keys are in the ~/.docker/trust directory; safeguard and

Initialize a new Docker Swarm (you may want to do this on a VM):
back them up.
# docker swarm init --advertise-addr
The DOCKER_CONTENT_TRUST is just an environment variable and will
die with your shell session. But trust validation should be implemented Create a file with some random text; this is your secret:
across the entire process, from the images building and the images
hosting in the registry to image execution in the nodes. # cat secret.txt
This is my secret
DOCKER CREDENTIALS AND SECRETS
Your software needs sensitive information to run: user password Create a new secret resource from this file:
hashes, server-side certificates, encryption keys, etc. This situation
# docker secret create somesecret secret.txt
is made worse by the nature of containers; you don’t just “set up a
server” — there’s a large number of distributed containers that may be
Create a Docker Swarm service with access to this secret; you can
constantly created and destroyed. You need an automatic and secure
modify the uid, gid, mode, etc.:
process to share this sensitive info.
# docker service create --name nginx --secret
CORE PR ACTICES: source=somesecret,target=somesecret,mode=0400 nginx
•   Do not use environment variables for secrets; this is a very
common yet very insecure practice. Log into the Nginx container; you will be able to use the secret:

•   Do not embed any secrets in the container image. For example,

CODE BEGINS ON NEXT PAGE
GitGuardian published a report about secrets in popular Docker

REFCARD | FEBRUARY 2022 4 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | DOCKER SECURIT Y

root@3989dd5f7426:/# cat /run/secrets/somesecret - containerPort: 80

# This is my secret env:
root@3989dd5f7426:/# ls /run/secrets/somesecret - name: SOME_ENV_VAR
-r-------- 1 root root 19 Aug 28 16:45 /run/secrets/ valueFrom:
somesecre secretKeyRef:
name: secret-name

This is a minimal proof of concept. At the very least, now your secrets key: secret-key

are properly stored and can be revoked or rotated from a central point
of authority.
HASHICORP VAULT EX AMPLE
You can use different secret providers. Hashicorp Vault allows you to
store SSL Certificates, passwords, and tokens. It supports symmetric
and asymmetric keys. To use it in Docker, you need to pull the official
Hashicorp Vault image from the Docker Hub. Then you can run the Vault
development server using the following command:
$ docker run --cap-add=IPC_LOCK -d --name=dev-vault
vault
To run vault in server mode for non-dev environments, you can use the
following command:
$ docker run --cap-add=IPC_LOCK -d --name=dev-vault
vault
CONTAINER RESOURCE ABUSE
Containers are much more numerous than virtual machines on average.
They are lightweight and you can spawn big clusters of them on modest
hardware. Software bugs, design miscalculations, or a deliberate
malware attack can easily cause a Denial of Service if you don’t properly
configure resource limits.
To add to the problem, there are several different resources to
safeguard: CPU, main memory, storage capacity, network bandwidth,
I/O bandwidth, swapping, etc. There are some kernel resources that are
not so evident, and even more obscure resources such as user IDs (UIDs)
exist.
CORE PR ACTICES:
Limits on these resources are disabled by default on most
containerization systems; configuring them before deploying to
production is basically a must. There are three fundamental steps:

As an alternative to the Hashicorp Vault, you can use Azure Key Vault 1. Use the resource limitation features bundled with the Linux
and environment variables to pass secrets to the Docker container. In kernel and/or the containerization solution.
AWS, you can use secrets manager. To pass parameters to a Docker 2. Try to replicate the production loads on pre-production. Some
container, you can use AWS SDK. For example: people use synthetic stress tests, and others choose to "replay"

aws --profile <YOUR PROFILE FROM ~/.aws/credentials> the actual real-time production traffic. Load testing is vital to
--region <REGION eg. us-east-1> secretsmanager get- knowing where the physical limits are and where your normal
secret-value --secret-id <YOUR SECRET NAME> range of operations is.

3. Implement Docker monitoring and alerting. You don’t want to

You can also use Kubernetes secrets and use it in the Docker container. hit the wall if there is a resource abuse problem. Malicious or
You can achieve this by creating an environment variable with the not, you need to set thresholds and be warned before it’s too
secretKeyRef section.
late.

apiVersion: apps/v1
Example: Control groups, or cgroups, are a feature of the Linux kernel
kind: Deployment
that allow you to limit the access processes and containers have to
metadata:
system resources. We can configure some limits directly from the
name: back-end-project
spec: Docker command line, like so:
replicas: 1
# docker run -it --memory=2G --memory-swap=3G ubuntu
selector:
bash
matchLabels:
app: back-end-project
template: This will limit the container to 2GB main memory, 3GB total (main +

spec: swap). To check that this is working, we can run a load simulator; for
containers: example, the stress program present in the Ubuntu repositories:
- name: back-end-project
root@e05a311b401e:/# stress -m 4 --vm-bytes 8G
image: <path to docker container>
ports:

CODE CONTINUES IN NEXT COLUMN

REFCARD | FEBRUARY 2022 5 BROUGHT TO YOU IN PARTNERSHIP WITH
 REFCARD | DOCKER SECURIT Y

You will see a "FAILED" notification from the stress output. If you tail flaw was fixed long ago upstream, but not in your local image. This
the syslog on the hosting machine, you will be able to read something kind of problem can go unnoticed for a long time if you don’t take the
similar to: appropriate measures.

Aug 15 12:09:03 host kernel: [1340695.340552] Memory CORE PR ACTICES:

cgroup out of memory: Kill process 22607 (stress)
Picturing the containers as immutable atomic units is really nice for
score 210 or sacrifice child
architecture design, but from the security perspective, you need to
Aug 15 12:09:03 host kernel: [1340695.340556] Killed
regularly inspect their contents:
process 22607 (stress) total-vm:8396092kB, anon-
rss:363184kB, file-rss:176kB, shmem-rss:0kB •  Update and rebuild your images periodically to grab the newest
security patches. Of course, you will also need a pre-production
Using Docker stats, you can check current memory usage and limits. testbench to make sure these updates are not breaking
If you are using Kubernetes, you can actually book the resources that production.
your application needs to run properly and define maximum limits
•  Live-patching containers is usually considered a bad practice.
using requests and limits on each pod definition. Figure 1 below
The pattern is to rebuild the entire image with each update.
demonstrates the algorithm:
Docker has declarative, efficient, easy-to-understand build

Figure 1 systems, so this is easier than it may sound at first.

•  Use software from a distributor that guarantees security

updates. Anything you install manually out of the distro, you
have to manage security patching yourself.

•  Docker and microservice-based approaches consider

progressively rolling over updates without disrupting uptime,
which is a fundamental requisite of their model.

•  User data is clearly separated from the images, making this

whole process safer.

•  Keep it simple. Minimal systems expect less frequent updates.

Remember the intro: Less software and moving parts equals
less attack surface and updating headaches. Try to split your
containers if they get too complex.

•  Use a vulnerability scanner. There are plenty out there, both free
and commercial. Try to stay up to date on the security issues
READ-ONLY MODE FOR FILESYSTEM AND of the software you use subscribing to the mailing lists, alert
VOLUMES services, etc.
You should enable Read-only mode to restrict usage of filesystem or
•  Integrate this vulnerability scanner as a mandatory step of your
volumes. To achieve this, you can use a --read-only flag. For example:
CI/CD and automate where possible; don’t just manually check

docker run --read-only back-end the images now and then.

Example: There are multiple Docker image registry services that offer
You can also temporarily enable access if your app needs to save some
image scanning. For this example, we decided to use CoreOS Quay,
data. You can use the flag --tmpfs:
which uses the open-source Docker security image scanner Clair. While

docker run --read-only --tmpfs back-end Quay is a commercial platform, some services are free to use. You can
create a personal trial account by following these instructions.

STATIC VULNERABILITY SCANNING

Once you have your account, go to Account Settings and set a new
Containers are isolated black boxes: If they are doing their work as
password (you need this to create repos).
expected, it’s easy to forget which software and version is specifically
running inside. Maybe a container is performing like a charm from Click on the + symbol on your top right and create a new public repo:
the operational point of view, but it’s running version X.Y.Z of the
SEE IMAGE ON NEXT PAGE
web server, which happens to suffer from a critical security flaw. This

REFCARD | FEBRUARY 2022 6 BROUGHT TO YOU IN PARTNERSHIP WITH


We go for an empty repository here, but you have several other options,
as you can see in the image above.
Now, from the command line, we log into the Quay registry and push a
local image:
# docker login quay.io
# docker push quay.io/<your_quay_user>/<your_quay_
image>:<tag>
Once the image is uploaded into the repo, you can click on its ID
and inspect the image security scan, ordered by severity, with the
associated CVE report link and upstream patched package versions.
You can find additional detail in the Lab 2 “Push the example container
images to a container image registry.”
Alternatively, you can use tools like:
•  Harbor open-source registry with vulnerability scanning. Harbor
is based on policy and RBAC access rules. Also, Harbor has an
integration with other OIDC providers.
•  Trivy is an open-source tool that checks your images,
dockerfiles, and even IaC scripts (including Kubernetes and
Terraform) for vulnerabilities. Trivy is quite easy to use. Install
the binary file and run the following command:
$ trivy image back-end-app:1.2.2
REFCARD | FEBRUARY 2022
REFCARD | DOCKER SECURIT Y
If you use Kubernetes, you can combine one of these tools with
kubebench. Therefore, you will cover not only the containers side of
things but Kubernetes configuration security as well.
RUNTIME CONTAINER SECURITY
DOCKER INTRUSION DETECTION
In the previous sections, we covered the static aspects of Docker
security: vulnerable kernels, unreliable base images, and capabilities
that are granted or denied at launch-time, etc. But what if, despite all
these, the image has been compromised during runtime and starts to
show suspicious activity?
All the previously described static countermeasures do not cover all
attack vectors. What if your own in-house application has a vulnerability?
Or attackers are using a 0-day not detected by the scanning? Runtime
security can be compared to Windows anti-virus scanning: Detect and
prevent an existing break from further penetration.
CORE PR ACTICES:
•  Do not use runtime protection as a replacement for any other
static up-front security practices: Attack prevention is always
preferable to attack detection. Use it as an extra layer of peace-
of-mind.
•  Having generous logs and events from your services and hosts
— correctly stored, easily searchable, and correlated with any
change you do — will help a lot when you have to do a post-
mortem analysis.
There are several tools that can help you to prevent this issue:
Wazuh, a free open-source system that helps to detect and prevent
threats, not only contains integrated intrusion detection but also log
analysis, log data analytics, vulnerability detection, cloud security, and
many other security features. Wazuh has an agent that is integrated
into the Docker environment. The Agent helps to detect hidden files,
unregistered network listeners, and cloaked processes. Wazuh contains
the following components:
•  Server
•  Elastic stack
•  Agent
It also supports platforms like Kubernetes, Chef, Puppet, and AWS
CloudFormation.
Let’s test and deploy Wazuh in an example Linux environment. To do
this, perform the following actions:
sudo apt-get update
sudo apt-get installcurl apt-transport-https lsb-
release gnupg2
CODE CONTINUES ON NEXT PAGE
7 BROUGHT TO YOU IN PARTNERSHIP WITH
 REFCARD | DOCKER SECURIT Y

of your infrastructure, depending on the specific service that image is

wget -qO - https://artifacts.elastic.
providing to an application — think load balancer images like HAProxy
co/GPG-KEY-elasticsearch | sudo apt-key add -
or Nginx.
echo "deb https://artifacts.elastic.co/packages/7.x/
apt stable main" | sudo tee -a /etc/apt/sources.
Often, teams apply standard security policies to an image and don’t
list.d/elastic-7.x.list
differentiate at the application and orchestration layer, which can
sudo apt-get update
leave holes in the service by trying to protect everything via blanketed
policies. This is where tight integrations with an orchestrator and the
Let’s install the Wazuh manager, its API, and NodeJS:
metadata they provide is needed to differentiate between the services
sudo apt-get install wazuh-manager that are running an image.
service wazuh-manager status
sudo curl -sL https://deb.nodesource.com/setup_8.x | CORE PR ACTICES:
sudo bash - •   Differentiate between images’ base of orchestration and
sudo apt-get install nodejs containers’ labels for more granular policies.
sudo apt-get install wazuh-api
•   Take actions like killing or pausing a container when a policy
sudo service wazuh-api status
has been violated. Often, the orchestrator will spin up a new
container, thus bringing the environment to a safe state.
DETECTING AND BLOCKING ATTACKS ON
CONTAINERS IN THE CLOUD ENVIRONMENT •   Rely on orchestration metadata rather than trying to manually
All leading cloud providers offer services that secure your container on update container image hashes.
the different layers. For example:
Example: Let's look into preventing data exfiltration in Kubernetes
•   Secure container registries with Istio. Istio is a platform that provides security opportunities for:
•   Secure Kubernetes setup •   Containers

Azure has a service called Azure Defender. It can scan containers in the •   Kubernetes
Azure Container Registry and in Azure Kubernetes Service. •   Infrastructure as a Code

Azure Defender for Docker containers in the Azure registries has several •   Network
triggers. For example, containers can be scanned when Docker images
First, you should install Minikube (or use cloud services like AKS)
are pushed to your registry. Below, you can see an image with the Azure
and Istio. Then follow this step-by-step guide from the IBM example
Defender Analysis report:
repository.

INCIDENT RESPONSE
POST-MORTEM ANALYSIS
Incident response becomes harder in container environments because
of their ephemeral nature. A bad actor can delete a container after an
attack to remove all traces of any file access or network activity.

CORE PR ACTICES:
•   Capture network, file, system call, and user data from inside the
container around any policy violation.

•   Commit containers that have policy violations so that they can

be examined and ran outside of production.

•   Use JWT token-based authentication for a Docker registry.

DETECTING AND BLOCKING ATTACKS ON •   Set up additional security policies with the admission controller

ORCHESTRATED MICROSERVICES and Open Policy Agent

Containers are the base building block of a service that is managed

Example: Forensic analysis with the docker-forensics-toolkit. This is
by popular orchestrators like Kubernetes or Docker Swarm. There
an open-source toolkit that performs analysis in your Docker
are many cases where the same image will be used in different areas
environment based on HDD copies of the host. To test this tool:

REFCARD | FEBRUARY 2022 8 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | DOCKER SECURIT Y

Run the following cmdlet to mount the host image:

sudo python src/dof/main.py mount-image testimages/
alpine-host/output-virtualbox-iso/packer-virtualbox-
iso-*-disk001.vmdk.raw
Then, execute tests with the following command:
sudo pytest --image-mountpoint=/tmp/test-4-root-2
WRITTEN BY BORIS ZAIKIN,
SOFTWARE AND CLOUD ARCHITECT AT NORDCLOUD
GMBH
I'm a Certified Software and Cloud Architect who
has solid experience designing and developing
complex solutions based on the Azure, Google, and AWS clouds.
I have expertise in building distributed systems and frameworks
based on Kubernetes and Azure Service Fabric. My areas of interest
include enterprise cloud solutions, edge computing, high load
applications, multitenant distributed systems, IoT solutions.

This allows you to

•  Gather information, metadata, and history for all images found

on the PC
•  Show image and container configuration files
•  Print container log files
•  Mount the container file system to any given location
600 Park Offices Drive, Suite 300
Research Triangle Park, NC 27709
888.678.0399 | 919.678.0300
At DZone, we foster a collaborative environment that empowers developers and
tech professionals to share knowledge, build skills, and solve problems through
content, code, and community. We thoughtfully — and with intention — challenge
the status quo and value diverse perspectives so that, as one, we can inspire
positive change through technology.

CONCLUSION
Copyright © 2022 DZone, Inc. All rights reserved. No part of this publication
In this Refcard, we’ve walked through the core practices of security may be reproduced, stored in a retrieval system, or transmitted, in any form or
by means of electronic, mechanical, photocopying, or otherwise, without prior
before containers even enter production environments all the way written permission of the publisher.
up to the more complex analysis of a rootkit installation. We’ve used
components baked into the Docker runtime, as well as open-source
tools like Quay, Forensics-toolkit, and Istio to analyze real-world use
cases such as greedy containers or mapping network communication.

As you can see, Docker security can start very simply yet grow more
complex as you take containers into production. Get experience early
and then grow your security sophistication to what your environment
requires.

REFCARD | FEBRUARY 2022 9 BROUGHT TO YOU IN PARTNERSHIP WITH