#CLUS

A comprehensive look
at security within the
Cisco Container
Platform
Sanjeev Rampal
Principal Engineer, Cloud Platforms BU
BRKCLD-2011

#CLUS
Agenda
• Introduction to Cisco Container Platform
• Security Model, Agile delivery, Sample Topology
• Platform Hardening & Cisco Secure Development
• Kubernetes & Container Security
• Kubernetes Secure Multi-tenancy
• Demo

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot# BRKCLD-2011

by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Can a look at security ever be “comprehensive” ?

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Cisco Container Platform Architecture
Control Plane Data Plane

Workloads
Workloads

Cluster 1

Cluster 2
Cluster 1

Cluster 2
Pod Pod

Orchestration
Automation

Operations

Ops

Ops
Pod Pod

Pod Pod
HX Connect Cluster/
Machine
Controllers Cluster 1 Kubernetes Cluster 2 Kubernetes

Control Plane Kubernetes VM VM VM VM VM VM

VM VM VM VM

Storage (HyperFlex / VMware) Hypervisor Layer (HyperFlex / VMware)

Compute Hardware (UCS)

Networking (Nexus 9K)

Kubernetes Fluentd Prometheus Kibana Hyperflex Contiv Istio

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Ops Personas & Logical production layout
Tenant cluster 1
CCP Admin (IT Ops) Devops admin/ Dev
CCP api, RBAC Add-ons K8s api, RBAC
K8s data plane
K8s

CCP admin Ubuntu

CCP app

K8s
Tenant cluster 2
Devops Admin/ Dev
Web based Ubuntu Add-ons
Installer VM
K8s api, RBAC
K8s K8s data plane

Full cluster & services Ubuntu

life-cycle mgmt
“Immutable” infra
#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Security Model,
Agile delivery,
Sample Topology
 Software Layering & CCP Security Scope
End-user Application
responsibility

End-user
CCP Application Addons Addons
Applications
CCP
packaging & Kubernetes, Docker, Container Kubernetes, Docker, Container
Security infra plugins infra plugins
responsibility

VMs, Instances, Node OS VMs, Instances, Node OS

Physical Infra Hypervisor, Virtualization infra Hypervisor, Virtualization infra

separate e.g. vSphere e.g. vSphere
setup +
responsibility Physical Compute, Network, Physical Compute, Network,
Storage Storage

Control cluster Tenant cluster

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Agile Delivery & Immutable infra based model
• Immutable Infrastructure
• Integrated provisioning and full lifecycle management of of infrastructure (VMs, Node OS etc)
along with Kubernetes, container infra
• No additional software patching or maintenance needed for Node OS
• Centralized upgrades + patching of combined infra => No configuration drift or snowflakes

• Continuous Release and Delivery

• Bi-weekly internal releases, Monthly external releases, patch releases asap when needed

=> Improved overall product security, predictability & quick

turnaround of security patches

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 Sample secure deployment: Private stub network
Inbound
Private stub network Proxy IP gateways
w/ RFC 1918 addressing

IPAM External
K8S cluster Routed n/w

SNAT

Outbound
Proxy (optional)
Firewall
vSphere
cluster K8S Cluster Exposed k8s api
Pod IPs Node IPs and application CCP Non-
192.168.0.0/16 IPs/ VIPs containerized
control Oracle DB (for
K8S cluster example)
Service IPs
10.96.0.0/12

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Multi-cloud deployment: Cisco CP + AWS EKS

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Platform
Hardening & Cisco
Secure
Development
CCP Platform Hardening incl CSDL
• Cisco Container Platform is developed using the comprehensive
security requirements defined in the Cisco Secure Development
Lifecycle (CSDL) process
• Curated Ubuntu OS from Canonical
• Cisco performs additional hardening of containers (internally
developed for CCP application as well as sourced from upstream)
• Frequent internal vulnerability scanning & fixing of every CCP
release using a mix of external vendor container security tools as
well as internal tooling

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 BOM created, BOM to IPC
reviewed and
approved
IPC
Ubuntu, K8S …

To CCO

Releases into CCO

Developed code
CCP CI file repo
Release built

(CCP Github)
CCP CI registry

Container artifacts
Ex. Prometheus, On-demand test deployment
Requirements NGINX etc

Run-time test
Input Static registry
scan
+
Vulnerability
Vulnerability alert
Scanning tools
feeds

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Additional platform hardening features
• TLS communication for http traffic (Encrypted data in motion
internal and external)
• Support for TLS 1.3 on CCP API/Dashboard
• Strong ciphers for internal encrypted data at rest
• ecdsa and ed25119 keys for ssh into cluster nodes
• Continuous monitoring of NVD and industry standard vulnerability
intelligence streams and rapid turnaround of patch releases
• Recent industry CVEs fixed & delivered rapidly on CCP
• Example: Critical k8s patch delivered in 2 weeks … CVE-2018-1002105: proxy
request handling in kube-apiserver can leave vulnerable TCP connections

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
CCP App Security: Role Based Access Control

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Kubernetes &
Container Security
Kubernetes Security is a Journey

From: J Jalava: Kubernetes Security Journey

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Tenant Kubernetes & Docker Security in CCP
K8S Security related features on CCP Kubernetes clusters
• K8S dashboard protection
• K8S Authentication
• K8S Authorization
• K8S Cert manager
• K8S Encrypted secrets
• Kubernetes Ingress with TLS/ https
• Istio Ingress gateway + Service mesh
• K8S Network policy
• Secure Multi-tenancy, Admission controllers, Pod Security Policies, AppArmor, Kata (future)

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 K8S dashboard protection
Kubernetes dashboard locked in CCP

"The hackers had infiltrated Tesla's Kubernetes console

which was not password protected," - ArsTechnica

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
K8S AuthN, AuthZ, Admission Control flow

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
K8S AuthN, AuthZ on CCP K8S tenant clusters
• K8S Authentication options:
• X.509 Client certificates <Suggested for simple deployments only>

• If team has AWS account, can use AWS IAM with on-prem CCP K8S
• Integrate 3rd party identity solutions e.g. Tremolo
• Direct Kubernetes OIDC-LDAP integration (future)

• K8S Authorization options:

• ABAC: Disabled on CCP K8S
• RBAC: Role Based Access Control; enabled by default on CCP K8S
• Authorization webhooks (Tech preview; full support in upcoming release)
• Open Policy Agent (Tech preview; full support in upcoming release)

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
AWS IAM Authentication for On-prem CCP K8S

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Kubernetes cert-manager & encrypted secrets
• Kubernetes Cert-manager:
• Kubernetes project to automate generation of X.509 certificates
• Used in CCP to generate certs for internal communication & external API

• Kubernetes Encrypted secrets:

• Kubernetes state in etcd can be encrypted using Kubernetes encrypted
secrets feature
• Note, this feature must be enabled via CCP api, not exposed to GUI yet
• Set etcd_encrypted=True to enable this capability per tenant k8s cluster

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Admission Controllers & Secure Multi-tenancy
• New feature (Tech preview in CCP v3.2)
• Setting secure_multitenancy_enabled to True enables
1. Multiple built-in k8s admission controllers on the new cluster:
• PodSecurityPolicy
• LimitRanger
• ResourceQuota
• ValidatingAdmissionWebhook
• MutatingAdmissionWebhook

2. Privileged & restricted pod security policies and associated

PodSecurity and RBAC policies and bindings (with AppArmor and
Seccomp based tenant and container isolation)
3. Privileged-tenant & restricted-tenant as sample tenants

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Kubernetes Network Policies Support
• Supported on CCP tenant clusters
App-front
App-front for all 3 CNI options
• Network microsegmentation tool
within Application + across teams
• L3, L4 CNI network policies (ingress
App-core1 App-metrics
and egress)
• Extra network policy options when
using ACI CNI

App-db • L7 policies on K8S Ingress (Nginx)

and Istio (Envoy)
• E-W http traffic encryption w/ Istio

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Kubernetes Network Policies on CCP+ACI CNI
Network Policy
Kubernetes Technical Description
• Network policies of Kubernetes supported using standard
upstream format but enforced through OpFlex / OVS using
APIC Host Protection Profiles
ACI Policies
• Kubernetes app configurations can be moved without
modification to/from ACI and non-ACI environments
• Standard K8S Container Network policies + (optional)
enhanced ACI container network policies

OpFlex OVS OpFlex OVS

Node Node

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Demo: Secure Multi-tenancy in CCP K8S
More Information
• https://www.cisco.com/c/en/us/products/cloud-systems-
management/container-platform/index.html
• https://www.cisco.com/c/en/us/support/cloud-systems-
management/container-platform/tsd-products-support-series-
home.html
• http://www.cisco.com/go/multicloud
• Webex space for this session cs.co.ciscolivebot#BRKCLD-2011
• Or contact/ follow: srampal@cisco.com @sr2357

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Conclusion: Cisco Container Platform
.. so that you won’t need to be this guy

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Thank you

#CLUS
#CLUS
Backup content
K8S Security: Some key standards & initiatives
• CIS Docker Benchmark https://www.cisecurity.org/benchmark/docker/
• CIS Kubernetes Benchmark https://www.cisecurity.org/benchmark/kubernetes/
• NIST SP 800-190 https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf
• AppArmor http://wiki.apparmor.net/index.php/Main_Page
• SELinux https://selinuxproject.org/page/Main_Page
• CRI-O https://github.com/cri-o/cri-o
• Kata containers https://katacontainers.io/
• And more …
• These are in addition to common infrastructure security & compliance related standards such as
Common Criteria, FIPS, PCI-DSS, GDPR, HIPAA

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 Preconfigured Pod Security Policy: “restricted”
apiVersion: policy/v1beta1 hostNetwork:false
kind: PodSecurityPolicy
metadata: hostIPC:false
name:restricted hostPID:false
annotations:
runAsUser:
seccomp.security.alpha.kubernetes.io/allowedProfileNa rule:'MustRunAsNonRoot’
mes:'docker/default’
apparmor.security.beta.kubernetes.io/allowedProfileNam seLinux:
es:'runtime/default’ rule:'RunAsAny’
seccomp.security.alpha.kubernetes.io/defaultProfileNam
e:'docker/default’ supplementalGroups:
apparmor.security.beta.kubernetes.io/defaultProfileNam rule:'MustRunAs’
e:'runtime/default’
spec: ranges:
privileged:false - min:1
allowPrivilegeEscalation:false
requiredDropCapabilities: max:65535
-ALL fsGroup:
volumes:
-'configMap’ rule:'MustRunAs’
-'emptyDir’ ranges:
-'projected’
-'secret’
- min:1
-'downwardAPI’ max: 65535
-'persistentVolumeClaim'
readOnlyRootFilesystem:false

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Calico/ Contiv Overlay Container Networking
K8S master K8S compute K8S compute
nodes/ VMs 1..3 nodes/ VMs 1..M nodes/ VMs 1..M

Contiv
VXLAN overlays Non-contiv
VLAN traffic

VMWare VM Port group 100

Physical L3 gateways
#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 Secure On-Premises Deployment Topology

K8S-Blue
HX vSphere
Cluster CCP – CP1
PG20 10.1.2.0/24
K8S-Red

PG10 10.1.1.0/28

10.1.3.0/24
PG30

DHCP server* Leaf e.g. N93xx

(for pre-3.0 vCenter
releases)
ASR1K or any L3 GW

Spine e.g. N95xx 100.1.x.x

#CLUS BRKCLD-2011 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40