Kubernetes Architecture

Nigel Poulton

@nigelpoulton www.nigelpoulton.com
Kubernetes Architecture

Nigel Poulton

@nigelpoulton www.nigelpoulton.com
 Module Outline

Big picture view

Masters
Nodes
Pods
Services
Deployments
Recap
 Kubernetes
Big Picture View
 Team

Manager
(coach)
Team
 Team

RW HM
CH
S RB
Manager LW
(coach)
CH AM
LB
S
S
S1 GK
1

S2 S3
 HTTPS HTTPS

Search Auth

K/V store MySQL

Log Load
balancer
 HTTPS HTTPS

Search Auth

K/V store MySQL

Log Load
balancer
 Load
balancer

HTTPS
HTTPS

Auth

Search Log

Node 1

Node 2

Node 3 K/V store

MySQL
 Node Node

Master Node Node

Master
Master

Node Node

K8s cluster

Nodes a.k.a. Minions

 Node Node

Master Node Node

Master
Master

Node Node

K8s cluster

Nodes a.k.a. Minions

 Node Node

Master Node Node

Master
Master

Node Node

K8s cluster

Nodes a.k.a. Minions

Master

KV

apiserver {}
 Node Node
Pod

Master Node Node

Master
Master Pod

Node Node
Pod Pod

K8s cluster

Nodes a.k.a. Minions

 Masters
The Kubernetes Control Plane
 Master
Linux

Masters Fe

The Kubernetes Control Plane

…
Node Node Node
Linux Linux Linux
Fe Fe Fe
 Master

KV

{}

Linux
Fe
 Multi-master HA

Master
Master
KV
Master
KV
KV

{}
{}
Linux {}
Fe Linux
Fe Linux
Fe
Node Node
Linux Linux
Fe Fe

Node Node
Linux Linux
Fe Distributed Control Plane Fe
(future)
 Pod
Pod Pod

Node Node
Linux Linux
Fe Fe

Pod
Pod Pod

Node Node
Linux Linux
Fe Don’t run user workloads on Fe
“Master”
 Pod
Pod Pod

Node Node
Linux
Master Linux
Fe Fe
KV

{}

Pod Linux
Pod Pod
Fe
Node Node
Linux Linux
Fe Don’t run user workloads on Fe
“Master”
 Master

KV

apiserver
{} {}

Linux
Fe
 Master

KV

apiserver {}

Linux
Fe
kube-apiserver Master

KV
Front-end to the control
plane
Exposes the API (REST)
apiserver {}
Consumes JSON
(via manifest files)
Linux
Fe
kube-apiserver Master

KV
Front-end to the control
plane
Exposes the API (REST)
apiserver {}
Consumes JSON
(via manifest files)
Linux
Fe
Cluster store
Master
Persistent storage Cluster
KV Store
KV
Cluster state and config
Uses etcd
Distributed, consistent,
watchable… apiserver {}

The “source of truth” for Linux

the cluster Fe
Have a backup plan for it!
Cluster store
Master
Persistent storage Cluster
KV Store
KV
Cluster state and config
Uses etcd
Distributed, consistent,
watchable… apiserver {}

The “source of truth” for Linux

the cluster Fe
Have a backup plan for it!
kube-controller-manager
Controller of controllers Master
• Node controller Cluster
KV Store
KV
• Endpoints controller
• Namespace controller Controller
• …
Watches for changes apiserver {}

Helps maintain desired

Linux
state
Fe
kube-controller-manager
Controller of controllers Master
• Node controller Cluster
KV Store
KV
• Endpoints controller
• Namespace controller Controller
• …
Watches for changes apiserver {}

Helps maintain desired

Linux
state
Fe
kube-scheduler Master
Cluster
KV Store
KV
Watches apiserver for
new pods
Scheduler Controller

Assigns work to nodes

• affinity/anti-affinity apiserver {}

• constraints
Linux
• resources
Fe
• …
kube-scheduler Master
Cluster
KV Store
KV
Watches apiserver for
new pods
Scheduler Controller

Assigns work to nodes

• affinity/anti-affinity apiserver {}

• constraints
Linux
• resources
Fe
• …
 Master
Cluster
KV Store
KV

Scheduler Controller

apiserver {}

Linux
Fe
Master
Cluster
KV Store
KV

Scheduler Controller

apiserver {}
Master
Cluster
KV Store
KV Pod Pod

Node
Scheduler Controller

apiserver {}
commands

Pod Pod

Node

$kubectl JSON
 Nodes
a.k.a. “Minions”
The Kubernetes Workers
 Node

Nodes
K
a.k.a. “Minions”
The Kubernetes Workers

Linux
Fe
 Node

K
Linux
Fe
 K
Node
Kubelet
• The main Kubernetes agent
•
•
•
•
Registers node with cluster
Watches apiserver
Instantiates pods
Reports back to master
K
• Exposes endpoint on :10255
Linux
Fe
 Node
Container Engine
Does container management:
• Pulling images
• Starting/stopping containers
• …
K
Pluggable:
• Usually Docker Linux
• Can be rkt Fe
 Node
kube-proxy
Kubernetes networking:
• Pod IP addresses
• All containers in a pod
share a single IP
K
• Load balances across all
pods in a service
Linux
Fe
 Kubelet
K Main Kubernetes agent
Node

K
Linux
Fe
 Kubelet
K Main Kubernetes agent
Node

Container engine K
Docker or rkt

Linux
Fe
Pod Pod Pod Pod

front-end front-end front-end front-end

Service
IP: 10.0.0.60
DNS: db.myservice

Pod Pod Pod

back-end back-end back-end

 Kubelet
K Main Kubernetes agent
Node

Container engine K
Docker or rkt

Linux
kube-proxy
Fe
Kubernetes networking
Declarative Model
&

Desired State
Master

KV

apiserver {}

Manifest
file

YAML or JSON
Describe desired
state
Master

KV

apiserver {}

3 x pods
nginx
…
 Master

KV

apiserver {}

3 x pods
nginx
…

Pod Pod Pod

nginx nginx nginx

Node Node Node
 Master
3 x pods
nginx
Desired state/
KV … record of intent
• 3 x nginx pods

apiserver {}

Actual state
• 3 x nginx pods

Pod Pod Pod

nginx nginx nginx

Node Node Node
 Master
3 x pods
nginx
Desired state/
KV … record of intent
• 3 x nginx pods

apiserver {}

Actual state
• 2 x nginx pods

Pod Pod Pod

nginx nginx nginx

Node Node Node
 Master
3 x pods
nginx
Desired state/
KV … record of intent
• 3 x nginx pods

apiserver {}

Actual state
• 2 x nginx pods

Pod Pod Pod Pod

nginx nginx nginx nginx

Node Node Node
 Master
3 x pods
nginx
Desired state/
KV … record of intent
• 3 x nginx pods

apiserver {}

Actual state
• 3 x nginx pods

Pod Pod Pod Pod

nginx nginx nginx nginx

Node Node Node
Pods
VM Container Pod

Atomic units of scheduling

VM Container Pod

Atomic units of scheduling

 Pod

Pod

Containers always run

inside of pods Pod

Pods can have multiple Pod

containers
(advanced use-case)
Pod
Ring-fenced environment
• Network stack Pod
• Kernel namespaces
• …
…
trnsprt
n containers
net
All containers in pod share link IPC mount …
the pod environment
 Tight Coupling

Pod
 Loose Coupling

Pod
 Loose Coupling

Pod Pod
 Loose Coupling

Pod Pod
 Pods and Scaling
Pod Pod

nginx nginx

Pod nginx

nginx
nginx

Pod
nginx
nginx
nginx
Pod

nginx
 Pods and Scaling
Pod Pod

nginx nginx

Pod nginx

nginx
nginx

Pod
nginx
nginx
nginx
Pod

nginx
 Multi-container Pods

Pod
Web server Log scraper

main sidecar
container container
 Multi-container Pods

Pod
Web server Log scraper

main sidecar
container container
Pods are Atomic

Pod
Env
net sidecar

IPC
mount
main
…
Pods are Atomic

Pod Pod
Env
net sidecar
Scale
IPC
mount
main
…
 Pods are Atomic

Pod Pod
Env Env
net sidecar net sidecar

IPC IPC
mount mount
main main
… …

#1 Status:ready #2 Status:pending
 Pods are Atomic

Pod Pod
Env Env
net sidecar net sidecar

IPC IPC
mount mount
main main
… …

#1 Status:ready #2 Status:pending
 Pod Lifecycle

Phase: pending Phase: running Phase: succeeded/failed

 Deploying Pods

Usually via higher level objects

 Master

KV

apiserver {}

apiVersion: v1
kind: Pod
metadata:

Pod

xyz
Node Node Node
 Deploying Pods

Usually via higher level objects

Deployment

Replication Controller

Pod

…
mnt
IPC
net ...
Services
 Node1 Node2 Node3
Pod Pod Pod Pod

fe db fe db

10.0.0.91 10.0.0.15 10.0.0.53 10.0.0.20

Node4 Node5 Node6

Pod Pod Pod Pod

db fe db fe

10.0.0.11 10.0.0.21 10.0.0.48 10.0.0.62

Every new pod gets a new IP = IP churn!

Pod Pod Pod Pod

front-end front-end front-end front-end

10.0.0.12 10.0.0.83 10.0.0.25 10.0.0.39

Pod Pod

db:v1 db:v1

10.0.0.91 10.0.0.44
Pod Pod Pod Pod

front-end front-end front-end front-end

10.0.0.12 10.0.0.83 10.0.0.25 10.0.0.39

Pod Pod

db:v1 db:v2
db:v1

10.0.0.91 10.0.0.44
10.0.0.70
Pod Pod Pod Pod

front-end front-end front-end front-end

10.0.0.12 10.0.0.83 10.0.0.25 10.0.0.39

Pod Pod Pod

db:v1 db:v2 db:v2

10.0.0.91 10.0.0.70 10.0.0.53

Pod Pod Pod Pod

front-end front-end front-end front-end

10.0.0.12 10.0.0.83 10.0.0.25 10.0.0.39

Pod Pod

db:v2 db:v2

10.0.0.70 10.0.0.53
Pod Pod Pod Pod

front-end front-end front-end front-end

10.0.0.12 10.0.0.83 10.0.0.25 10.0.0.39

Service
IP: 10.0.0.60
DNS: db.myservice

Pod Pod

db:v1 db:v1

10.0.0.91 10.0.0.44
Pod Pod Pod Pod

front-end front-end front-end front-end

10.0.0.12 10.0.0.83 10.0.0.25 10.0.0.39

Service
IP: 10.0.0.60
DNS: db.myservice

Pod Pod
10.0.0.37 10.0.0.100
db:v1 db:v1

10.0.0.61
10.0.0.91 10.0.0.44
Pod Pod Pod Pod

front-end front-end front-end front-end

10.0.0.12 10.0.0.83 10.0.0.25 10.0.0.39

Service
IP: 10.0.0.60
DNS: db.myservice

Pod Pod
10.0.0.37 10.0.0.100
db:v1
db:v1.2 db:v1.2
db:v1

10.0.0.61
10.0.0.113 10.0.0.44
10.0.0.32
 Service
Pod
IP: 10.0.0.60
auth
DNS: db.myservice
10.0.0.113

Pod Pod

db db

10.0.0.113 10.0.0.32
 Service
Pod
IP: 10.0.0.60
auth
DNS: db.myservice
10.0.0.113

Pod Pod

db db

10.0.0.113 10.0.0.32
Service
IP: 10.0.0.60
DNS: db.myservice
Service
IP: 10.0.0.60
DNS: db.myservice
 Only send to healthy pods

Service
IP: 10.0.0.60
DNS: db.myservice
 Only send to healthy pods
Can be configured for
session affinity
Service
IP: 10.0.0.60
DNS: db.myservice
 Only send to healthy pods
Can be configured for
session affinity
Service Can point to things outside
IP: 10.0.0.60 the cluster
DNS: db.myservice
 Only send to healthy pods
Can be configured for
session affinity
Service Can point to things outside
IP: 10.0.0.60 the cluster
DNS: db.myservice Random load balancing
 Only send to healthy pods
Can be configured for
session affinity
Service Can point to things outside
IP: 10.0.0.60 the cluster
DNS: db.myservice Random load balancing
Uses TCP by default
 Master

KV

apiserver {}

Deployments
Node Node Node Node
Pod Pod Pod Pod

xyz xyz xyz xyz

 Master

apiVersion: v1
kind: ReplicationController KV
metadata:
name: xyz
spec: apiserver {}
replicas: 4

Node Node Node Node

Pod Pod Pod Pod

xyz xyz xyz xyz

 Master

apiVersion: v1
kind: ReplicationController KV
metadata:
name: xyz
spec: apiserver {}
replicas: 4

Node Node Node Node

Pod Pod Pod Pod

xyz xyz xyz xyz

 Master

apiVersion: v1
kind: ReplicationController KV
metadata:
name: xyz
spec: apiserver {}
replicas: 4

Node Node Node Node

Pod Pod Pod Pod

xyz xyz xyz xyz

 Master

apiVersion: v1
kind: ReplicationController KV
metadata:
name: xyz
spec: apiserver {}
replicas: 4

Node Node Node Node

Pod Pod Pod Pod

xyz xyz xyz xyz

 Master

apiVersion: v1
kind: ReplicationController KV
metadata:
name: xyz
spec: apiserver {}
replicas: 4

Node Node Node Node

Pod Pod Pod Pod

xyz xyz xyz xyz

 Deployed via YAML or
REST objects Self documenting
JSON manifests

apiVersion: extensions/v1beta1
kind: Deployment Simple rolling updates
Spec-once deploy-many metadata: and rollbacks
name: xyz
spec:
replicas: 4

Add features to
Replication Controllers Deployed via the
Versioned
apiserver
(Replica Sets)
 Deployed via YAML or
REST objects Self documenting
JSON manifests

apiVersion: extensions/v1beta1
kind: Deployment Simple rolling updates
Spec-once deploy-many metadata: and rollbacks
name: xyz
spec:
replicas: 4

Add features to
Replication Controllers Deployed via the
Versioned
apiserver
(Replica Sets)
 Multiple concurrent versions
• Blue-green deployments
• Canary releases

apiVersion: extensions/v1beta1
kind: Deployment Simple rolling updates
metadata: and rollbacks
name: xyz
spec:
replicas: 4

Simple versioned rollbacks

Bringing it home!
K8s is all about orchestrating
containerized apps!
 Node Node

Linux Linux
Fe Fe
Master

KV

{}

Node Linux Node

Fe

Linux Linux
Fe Fe
 Node Node

Linux Linux
Fe Fe
Master (control plane)

KV

{}

Node Linux Node

Fe

Linux Linux
Fe Fe
 Node Node

Linux Linux
Master (control plane)
Fe Fe
Master (control plane)
KV
Master (control plane)
KV
KV
{}
{}
Linux
Fe {}
Node Linux Node
Fe Linux
Fe
Linux Linux
Fe Fe
 Node Node

Cluster Store
Linux Linux
Fe Fe
Master (control plane)

KV

{}

Node Linux Node

Fe

Linux apiserver Linux

Fe Fe
• Front-end to control plane
 Node Node
Cluster Store
• Cluster state and config
• Stateful
Linux Linux
Fe Fe
Master (control plane)

KV

{}

Node Linux Node

Fe

Linux apiserver Linux

Fe Fe
• Front-end to control
plane
 Node/Minion Node/Minion
Cluster Store
• Cluster state and config
• Stateful
Linux Linux
Fe Fe
Master (control plane)

KV
Kubelet
Main Kubernetes agent

{}

Container engine Linux Node/Minion

Docker or rkt
Fe

kube-proxy apiserver Linux

Fe
Kubernetes networking • Front-end to control
plane
 Master (control plane)
Node/Minion Node/Minion
KV
Pod

Linux xyz Linux

Fe Fe

{}

Linux
Fe

Node/Minion
Manifest
apiVersion: extensions..
kind: Deployment
metadata: Linux
name: xyz
Fe
spec:
replicas: 4
 Pods : Atomic unit of scheduling…

Replication
Objects
Controllers : Scale pods, desired state etc…
in the
K8s API
Deployments : RC + rolling updates, rollbacks…

Services : Stable networking…

 Coming up next…

Installing Kubernetes