Experimenting with Kubernetes

Ingress

1
 Anthony Macey
TOGAF certified architect with over 20 years of experience across public sector,
finance, health, oil, and telecoms. Keen pursuant of CPD and professional
member of the British Computing Society (BCS).
I have a special interests in data science, data integration, containers,
kubernetes and cloud infrastructure.

2
Synopsis
I don’t work for Oracle, any opinions expressed are my own and not that of Oracle or UKOUG. The content of this
talk is intended for information purposes only and is not a substitute for professional services. For all licencing
options I would refer you to your Oracle representative.

Intro Docker/Kubernetes CI/CD NGINX Ingress ISTIO Ingress Q&A

• Intro – Stats, Standards and Adoption.

• Docker/Kubernetes - High level kubernetes building blocks.
• CI/CD – Kubernetes CI/CD example.
• NGINX Ingress – High level view of NGINX ingress and use cases.
• ISTIO Ingress – High level view of Istio ingress and use cases.

3
Background

4
Open Container Initiative

Mission • The mission of the Open Container Initiative (OCI)

is to promote a set of common, minimal, open
standards and specifications around container
technology.

Key Principle’s • Not bound to higher level constructs such as a

particular client or orchestration stack.

• Not tightly associated with any particular

commercial vendor or project.

• Portable across a wide variety of operating

systems, hardware, CPU architectures, public
clouds, etc

5
Cloud Native Computing Foundation

Container packaged Running applications and processes in software

containers as an isolated unit of application
deployment, and as a mechanism to achieve
high levels of resource isolation. Improves
overall developer experience, fosters code and
component reuse and simplify operations for
cloud native applications.

Dynamically managed Actively scheduled and actively managed by a

central orchestrating process. Radically improve
machine efficiency and resource utilization while
reducing the cost associated with maintenance
and operations.

Micro-services Loosely coupled with dependencies explicitly

described (e.g. through service endpoints).
oriented Significantly increase the overall agility and
maintainability of applications. The
foundation will shape the evolution of the
technology to advance the state of the art
for application management, and to make
the technology ubiquitous and easily
available through reliable interfaces.

6
Docker
“Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code,
runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software
will always run the same, regardless of its environment.” http://www.docker.com

APP APP
APP APP APP APP
APP APP APP APP
Libs

Libs

Libs

Libs
Libs Libs

Linux OS Linux OS APP APP APP APP

Libs

Libs

Libs

Libs
Hypervisor Linux OS

Host Host

http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf
https://coreos.com/, https://www.projectatomic.io/

7
Orchestration

8
Adoption Statistics

https://www.datadoghq.com/docker-adoption/
https://www.cncf.io/blog/2017/06/28/survey-shows-kubernetes-leading-orchestration-platform/

9
De Facto Standard

https://sysdig.com/blog/2018-docker-usage-report/
https://www.theregister.co.uk/2017/10/17/docker_ee_kubernetes_support/
https://www.theregister.co.uk/2018/04/17/docker_enterprise_kubernetes/

10
Case Study

“After a year or so of using Mesos and Marathon, we decided to switch to

Kubernetes, which we run on a fleet of CoreOS machines. It is very tightly
integrated with Docker, and is the product of Google’s long experience running
containers in production at global scale. Deploying Kubernetes in a highly
available configuration on AWS is not for the faint of heart and requires you to
get familiar with its internals, but we are very pleased with the results. We
regularly kill hosts in production and Kubernetes quickly reschedules applications
to accommodate. In fact, we may soon deploy something like Netflix’s Simian
Army to deliberately induce failure and ensure our systems tolerate it
transparently.”

https://www.finextra.com/newsarticle/30789/monzo-pays-a-high-price-for-popularity-as-losses-widen-to-79-million
https://www.youtube.com/watch?v=YkOY7DgXKyw
https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/
https://techcrunch.com/2018/02/14/monzo-ireland/

11
Kubernetes
Build Blocks

12
Docker + Kubernetes
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized
applications. It groups containers that make up an application into logical units for easy management and discovery.
Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-
breed ideas and practices from the community.

• Automatic binpacking • Consistency, Resource QoS, Layers

• Self healing
• Horizontal scaling
• Automated rollouts & rollbacks
• Storage orchestration
• Batch execution
• Secret and configuration management
• Service discovery and load balancing
• + All the docker container benefits
(unionfs), Namespaces, Robustness,
Registry
• Everything @ Google: Runs in a container
and they start over 2 billion containers
per week. And as a result fewer than 100
people have root access to their vast
infrastructure.

13
Kubernetes // Namespaces

amacey@builder:~$ kubectl get pods -n hadoop-5jsiu

NAME READY STATUS RESTARTS AGE
hadoop-5jsiu-hadoop-hdfs-dn-0 1/1 Running 0 1d
hadoop-5jsiu-hadoop-hdfs-nn-0 1/1 Running 0 1d
hadoop-5jsiu-hadoop-yarn-nm-0 1/1 Running 0 1d
hadoop-5jsiu-hadoop-yarn-nm-1 1/1 Running 0 1d
hadoop-5jsiu-hadoop-yarn-rm-0 1/1 Running 0 1d

amacey@builder:~$ kubectl get pods -n website

NAME READY STATUS RESTARTS AGE
website-deployment-6c865b7c45-c8sr8 1/1 Running 0 2d

amacey@builder:~$ kubectl get pods

No resources found.

14
Kubernetes // Pods, Labels, Selectors, Services

apiVersion: v1 apiVersion: v1
kind: Pod kind: Service
metadata: metadata:
name: test-pod name: website-service
labels: spec:
app: website ports:
spec: - port: 80
containers: protocol: TCP
- name: private-reg-container selector:
image: tmdrg.azurecr.io/php71:v1 app: website
imagePullSecrets: type: NodePort
- name: tmdrg.azurecr.io

15
Kubernetes // Pods, Labels, Selectors & Services
amacey@builder:~/demo$ kubectl create -f shell-demo.yaml -n website
pod "shell-demo" created

amacey@builder:~/demo$ kubectl get pods -n website

NAME READY STATUS RESTARTS AGE
shell-demo 1/1 Running 0 33s
website-deployment-6c865b7c45-c8sr8 1/1 Running 0 2d

kubectl exec -it shell-demo -- /bin/bash -n website

root@shell-demo:/# apt-get update
root@shell-demo:/# apt-get install iputils-ping
root@shell-demo:/# ping website-service
PING website-service.website.svc.cluster.local (10.43.125.237) 56(84) bytes of data.

root@shell-demo:/# cat /etc/resolv.conf

nameserver 10.43.0.10
search website.svc.cluster.local svc.cluster.local cluster.local

16
Kubernetes // Pods, Labels, Selectors & Services
KUBE-DNS

Service (SELECTOR X) Service (SELECTOR Z)

POD A (LABEL X) POD B (LABEL X) POD A (LABEL Z) POD B (LABEL Z)

IMAGE A IMAGE A IMAGE B IMAGE B

KUBE-API

ETCD

17
Kubernetes // Secret Management
Objects of type secret are intended to hold sensitive information, such as passwords, OAuth tokens,
and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in POD POD
a pod definition or in a docker image. It allows you to change passwords and redeploy without
rebuilding the entire pipeline.

Dockerfile POD POD

Docker Image Image Registry Pull Deployment

Application Code

K8's

$ echo -n 'admin' > ./username.txt

$ echo -n '1f2d1e2e67df' > ./password.txt

$ kubectl create secret generic db-user-pass --from-file=./username.txt --from-

file=./password.txt
secret "db-user-pass" created

18
Kubernetes // Volumes & State

Service (SELECTOR X) Service (SELECTOR Y) Service (SELECTOR Z)

POD A (LABEL X) POD B (LABEL X) POD A (LABEL Y) POD B (LABEL Y) POD A (LABEL Z) POD B (LABEL Z)
IMAGE A IMAGE A IMAGE B IMAGE B IMAGE C IMAGE C

/var/volume/pod /var/volume/pod
NFS

19
Kubernetes // Volumes
metadata: kind: PersistentVolumeClaim
name: private-reg apiVersion: v1
spec: metadata:
containers: name: nginx-volume
- name: storageex spec:
image: nginx storageClassName: "oci"
ports: selector:
- name: web matchLabels:
containerPort: 80 oci-availability-domain: "PHX-AD-1"
volumeMounts: accessModes:
- name: nfs - ReadWriteOnce
mountPath: "/var/www/html" resources:
volumes: requests:
- name: nfs storage: 50Gi
nfs:
server: <HOSTNAME> volumes:
path: "/web_content" - name: nginx-storage
persistentVolumeClaim:
claimName: nginx-volume

20
Kubernetes // ConfigMap
$ kubectl create configmap hostcfg --from-file=/etc/hosts
configmap "hostcfg" created
volumeMounts:
$ kubectl describe configmap hostcfg - mountPath: /some_data
Name: hostcfg name: data
Namespace: default
- mountPath: /etc
Labels: <none>
Annotations: <none> name: config
volumes:
Data - name: data
====
hosts: emptyDir: {}
---- - name: config
127.0.0.1 localhost configMap:
10.1.0.25 builder.dk0.uk builder
name: hostcfg
# The following lines are desirable for IPv6 capable hosts items:
::1 localhost ip6-localhost ip6-loopback - key: hostcfg
ff02::1 ip6-allnodes path: hosts
ff02::2 ip6-allrouters

21
Kubernetes // Services & Ingress
NodePort; Exposes the deployment via a
Cluster NodePort Loadbalancer mesh network on a ephemeral port. If
Internal IP External Port/Mesh External LB
you destroy and recreate the service the
port changes.

ClusterIP; Creates an internal cluster ip

KUBE-PROXY
that’s discoverable via kube-dns and not
from outside the cluster.
IPTABLES

IPVS Rules *BETA*

Deployment LoadBalancer; Vendor implemented
external load balancer.

ExternalName; 1.7+ CNAME discoverable

service.

Pod Pod Pod Headless; This allows certain types of

application to control the discovery.

22
Kubernetes // Services & Ingress
An ingress controller is another way to expose an internal service externally, think virtual hosts and routing to backend
services. NGINX is a very typical tool used to implement an ingress controller. Essentially you define an ingress rule
that can contains all the information required to expose your service for example external DNS name, backend service
and any certificates.

Hostname

K8's Service
Certificates

Requests Ingress Controller Proxy Service/ClusterIP

23
Kubernetes // Deployments
apiVersion: apps/v1
kind: Deployment
metadata: replicas: 3
name: webapp-deployment
labels:
app: webapp
selector:
spec:
matchLabels:
replicas: 3
app: webapp
selector:
matchLabels:
app: webapp
template: livenessProbe:
metadata: httpGet:
labels: path: /
app: webapp port: 8001
spec: initialDelaySeconds: 15
containers: timeoutSeconds: 30
- name: webapp
image: tmdrg.azurecr.io/webapp:1
ports:
- containerPort: 8001
imagePullSecrets:
- name: regsecret

24
Kubernetes // Deployments
No

Are (n) Pods

Deployment Deploy (n) Pods Health Check
Deployed?

kube@k1:~/envs$ kubectl create –f webapp-deployment.yaml

Yes
kube@k1:~/envs$ watch kubectl get deployments

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

webapp-deployment 3 2 3 2 22s

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

webapp-deployment 3 3 3 3 7m

kube@k1:~/envs$ kubectl top nodes kube@k1:~/envs$ kubectl top pods

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% NAME CPU(cores) MEMORY(bytes)
k3 1385m 34% 2752Mi 17% webapp-deployment-6f4848c6d7-txwq6 3m 998Mi
k2 1310m 32% 3786Mi 23% webapp-deployment-6f4848c6d7-rtqld 3m 911Mi
k1 1471m 36% 2028Mi 12% webapp-deployment-6f4848c6d7-d56tl 3m 968Mi

25
Kubernetes // Types of Deployment
Deployment You describe a desired state in a Deployment object, and the Deployment controller changes the
actual state to the desired state at a controlled rate. You can define Deployments to create new
ReplicaSets, or to remove existing Deployments and adopt all their resources with new
Deployments.

StatefulSets Like a Deployment , a StatefulSet manages Pods that are based on an identical container spec.
Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods

DaemonSet Some typical uses of a DaemonSet are:

• running a cluster storage daemon, such as glusterd, ceph, on each node.

• running a logs collection daemon on every node, such as fluentd or logstash.
• running a node monitoring daemon on every node, such as Prometheus Node Exporter,
collectd, Datadog agent, New Relic agent, or Ganglia gmond.

CronJob One CronJob object is like one line of a crontab (cron table) file. It runs a job periodically on a
given schedule, written in Cron format.

26
CI/CD
Monitoring// Operations // Deployments

27
Kubernetes // Oracle Cloud Infrastructure

https://cloud.oracle.com/containers/kubernetes-engine
https://blogs.oracle.com/cloud-infrastructure/kubernetes-a-cloud-and-data-center-operating-system
https://app.wercker.com/

28
Kubernetes // Kubernetes as a Service

https://cloud.oracle.com/containers
https://www.youtube.com/watch?v=GFANezgZqCY
https://thenewstack.io/openworld-oracle-sets-sites-aws-new-serverless-
cloud-offerings/

29
Kubernetes // DevOps Tooling

30
Kubernetes // JIRA Driven Pipeline

31
Kubernetes // JIRA Driven Pipeline

https://www.datadoghq.com/docker-adoption/

32
NGINX Ingress
NGINX // cert-manager // Kubernetes

33
Kubernetes // Services & Ingress
An ingress controller is another way to expose an internal service externally, think virtual hosts and routing to backend
services. NGINX is a very typical tool used to implement an ingress controller. Essentially you define an ingress rule
that can contains all the information required to expose your service for example external DNS name, backend service
and any certificates.

Hostname

K8's Service
Certificates

Requests Ingress Controller Proxy Service/ClusterIP

https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-controllers 34
Ingress // NGINX Install

helm init

helm install --name cert-manager -f values.yaml --namespace kube-system \

--set rbac.create=false stable/cert-manager

helm install stable/nginx-ingress --set controller.service.externalTrafficPolicy=Local \

--set rbac.create=false --namespace kube-system

kubectl get service -l app=nginx-ingress --namespace kube-system

35
Ingress // NGINX
apiVersion: certmanager.k8s.io/v1alpha1 apiVersion: extensions/v1beta1
kind: Certificate kind: Ingress
metadata: metadata:
name: tls-service name: service.internal
spec: annotations:
secretName: tls-service kubernetes.io/ingress.class: nginx
dnsNames: certmanager.k8s.io/cluster-issuer: vault-issuer
- service.internal spec:
issuerRef: tls:
name: vault-issuer - hosts:
kind: ClusterIssuer - service.internal
secretName: tls-service
rules:
- host: service.internal
http:
paths:
- path: /api/v1/
backend:
serviceName: service.v1
servicePort: 80

https://docs.cert-manager.io/en/venafi/tutorials/vault/creating-vault-issuers.html

36
Ingress // NGINX
kube-system application

cert-
update Secret
manager

updates
Security Controls (Firewall)
Load balancer Frontend
nginx-
https://service.internal/api/v1/p1?value ingress- application
controller

maps
maps
nginx-
default- service
backend

https://github.com/jetstack/cert-manager
https://kubernetes.github.io/ingress-nginx/
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/ 37
Istio Ingress
Istio // Kubernetes

38
Ingress // Istio Install

helm init

kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml

kubectl apply -f install/kubernetes/helm/istio/charts/certmanager/templates/crds.yaml

helm template install/kubernetes/helm/istio --name istio --namespace istio-system > $HOME/istio.yaml

kubectl create namespace istio-system

kubectl apply -f $HOME/istio.yaml

39
Ingress // Istio Install

swiss@gcp:~$ kubectl get pods -n istio-system

NAME READY STATUS RESTARTS AGE
istio-citadel-7c4864c9d5-zsvf6 1/1 Running 0 5d
istio-cleanup-secrets-lxwrj 0/1 Completed 0 5d
istio-egressgateway-c7f44ff8-r9q6f 1/1 Running 0 5d
istio-galley-698f5c74d6-tvj5f 1/1 Running 0 4d
istio-ingressgateway-774d77cb7c-d5nns 1/1 Running 0 5d
istio-pilot-6bd6f7cdb-9lnrw 2/2 Running 0 5d
istio-policy-678bd4cf9-d55nh 2/2 Running 0 5d
istio-sidecar-injector-6555557c7b-p8smd 1/1 Running 0 5d
istio-telemetry-5f4cfc5b6-hnc5d 2/2 Running 0 5d
promsd-ff878d44b-7jxwx 2/2 Running 1 5d

40
Kubernetes // Istio Deployments
apiVersion: apps/v1 apiVersion: networking.istio.io/v1alpha3
kind: Deployment kind: VirtualService
metadata: metadata:
name: service-deployment name: service-deployment
labels: spec:
app: service-v1 hosts:
spec: - service.internal
replicas: 3 gateways:
selector: - service-gateway
matchLabels: http:
app: service-v1 - match:
template: - uri:
metadata: prefix: /service1
labels: route:
app: service-v1 - destination:
spec: host: service1.default.svc.cluster.local
containers: - match:
- name: service-v1 - uri:
image: tmdrg.azurecr.io/service:1 prefix: /service2
ports: route:
- containerPort: 8001 - destination:
imagePullSecrets: host: service2.default.svc.cluster.local
- name: regsecret - match:

41
Ingress // Istio

https://www.envoyproxy.io/
https://istio.io/docs/
https://istio.io/docs/tasks/ 42
Questions?

43
Scratch

44