Are they talking to each other?

Observing Kubernetes networking

with Cilium, Hubble and Grafana

Speaker: Anna Kapuścińska

Networking 101
in 3 slides
OSI model
OSI model
TCP UDP
Networking in Kubernetes
eBPF
Cilium
Hubble
eBPF Cilium Hubble
● “Plugins” for Linux kernel ● CNI plugin ● Observability layer for
● Use cases: ● L4 & L7 Network Policies Cilium
○ observability ● kube-proxy replacement ● CLI & UI
○ networking (load balancing) ● Prometheus metrics
○ security ● Ingress
● Service Mesh
eBPF for observability (not Hubble)

🚀 process default/xwing /usr/bin/vi /etc/passwd

📬 open default/xwing /usr/bin/vi /etc/passwd
📚 read default/xwing /usr/bin/vi /etc/passwd 1269 bytes
📪 close default/xwing /usr/bin/vi /etc/passwd
📬 open default/xwing /usr/bin/vi /etc/passwd
📝 write default/xwing /usr/bin/vi /etc/passwd 1277 bytes
💥 exit default/xwing /usr/bin/vi /etc/passwd 0
🚀 process default/xwing /usr/bin/curl http://cilium.io
🔌 connect default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
📤 sendmsg default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80 bytes 73
🧹 close default/xwing /usr/bin/curl tcp 10.244.0.6:34965 -> 104.198.14.52:80
💥 exit default/xwing /usr/bin/curl http://cilium.io 0
Hubble collects flows
Hubble exposes Prometheus metrics
metrics:
enabled:
- "drop"
- "flow:labelsContext=source_workload,destination_workload"
- "tcp"
- "kafka"
- "http"
- "dns:query;ignoreAAAA"
enableOpenMetrics: true
serviceMonitor:
enabled: true
dashboards:
enabled: true
Let’s debug a networking problem!
New demo app deployed to the cluster
$ curl localhost:8080

frontend: oh no, can't talk to worker 😢

sum(rate(hubble_dns_responses_total{rcode!="No Error"}[$__rate_interval])) by (rcode, destination_app)
$ hubble observe --to-namespace demo-app --to-label "app=frontend" --protocol dns

Oct 12 05:44:11.347: demo-app/frontend-5ffc945dcc-ngqcx:53083 (ID:14943) <- kube-system/coredns-6f4f8c7b4f-6kmd2:53 (ID:64997)

dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy worrker.demo-app.svc.cluster.local. A))
Oct 12 05:44:11.347: demo-app/frontend-5ffc945dcc-ngqcx:37887 (ID:14943) <- kube-system/coredns-6f4f8c7b4f-6kmd2:53 (ID:64997)
dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy worrker.demo-app.svc.cluster.local. AAAA))
Oct 12 05:44:11.347: demo-app/frontend-5ffc945dcc-ngqcx:39221 (ID:14943) <- kube-system/coredns-6f4f8c7b4f-6kmd2:53 (ID:64997)
dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy worrker.svc.cluster.local. AAAA))
Oct 12 05:44:11.347: demo-app/frontend-5ffc945dcc-ngqcx:34079 (ID:14943) <- kube-system/coredns-6f4f8c7b4f-6kmd2:53 (ID:64997)
dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy worrker.svc.cluster.local. A))
Oct 12 05:44:11.347: demo-app/frontend-5ffc945dcc-ngqcx:34902 (ID:14943) <- kube-system/coredns-6f4f8c7b4f-6kmd2:53 (ID:64997)
dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy worrker.cluster.local. AAAA))
Oct 12 05:44:11.347: demo-app/frontend-5ffc945dcc-ngqcx:58042 (ID:14943) <- kube-system/coredns-6f4f8c7b4f-6kmd2:53 (ID:64997)
dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy worrker.cluster.local. A))
...
$ kubectl -n demo-app edit deployment frontend
...
- env:
- name: WORKER_URL
value: http://worrker:8080 # <- typo
...

$ curl localhost:8080

frontend: oh no, can't talk to worker 😢

sum(rate(hubble_drop_total[$__rate_interval])) by (reason, source_app, destination_app)
$ kubectl -n demo-app edit cnp frontend
...
egress:
- toEndpoints:
- matchLabels:
app: worrker # <- typo
...

$ curl localhost:8080

frontend: oh no, can't talk to worker 😢

$ hubble observe --namespace demo-app --label "app=frontend" --verdict DROPPED

Oct 12 05:46:40.479: demo-app/frontend-5f47758b4c-ltk46:50014 (ID:14943) <> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)

policy-verdict:none DENIED (TCP Flags: SYN)
Oct 12 05:46:40.479: demo-app/frontend-5f47758b4c-ltk46:50014 (ID:14943) <> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
Policy denied DROPPED (TCP Flags: SYN)
Oct 12 05:46:41.487: demo-app/frontend-5f47758b4c-ltk46:50014 (ID:14943) <> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
policy-verdict:none DENIED (TCP Flags: SYN)
Oct 12 05:46:41.487: demo-app/frontend-5f47758b4c-ltk46:50014 (ID:14943) <> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
Policy denied DROPPED (TCP Flags: SYN)
Oct 12 05:46:43.507: demo-app/frontend-5f47758b4c-ltk46:50014 (ID:14943) <> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
policy-verdict:none DENIED (TCP Flags: SYN)
Oct 12 05:46:43.507: demo-app/frontend-5f47758b4c-ltk46:50014 (ID:14943) <> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
Policy denied DROPPED (TCP Flags: SYN)
(...)
sum(rate(hubble_tcp_flags_total{flag="SYN"}[$__rate_interval])) by (source_app) -
sum(label_replace(sum(rate(hubble_tcp_flags_total{flag="SYN-ACK"}[$__rate_interval])) by
(destination_app), "source_app", "$1", "destination_app", "(.*)")) without (destination_app)
$ hubble observe --namespace demo-app --label "app=frontend" --protocol tcp

Oct 12 06:06:05.907: demo-app/frontend-5f47758b4c-ltk46:55028 (ID:14943) -> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)

policy-verdict:L3-L4 ALLOWED (TCP Flags: SYN)
Oct 12 06:06:05.907: demo-app/frontend-5f47758b4c-ltk46:55028 (ID:14943) -> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
policy-verdict:L3-L4 ALLOWED (TCP Flags: SYN)
Oct 12 06:06:05.907: demo-app/frontend-5f47758b4c-ltk46:55028 (ID:14943) -> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
to-endpoint FORWARDED (TCP Flags: SYN)
Oct 12 06:06:06.920: demo-app/frontend-5f47758b4c-ltk46:55028 (ID:14943) -> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
to-endpoint FORWARDED (TCP Flags: SYN)
Oct 12 06:06:08.934: demo-app/frontend-5f47758b4c-ltk46:55028 (ID:14943) -> demo-app/worker-dcdc797d9-n9fj2:8000 (ID:15426)
to-endpoint FORWARDED (TCP Flags: SYN)
$ kubectl -n demo-app edit service worker
...
ports:
- port: 8080
protocol: TCP
targetPort: 8000 # <- mistake
...

$ curl localhost:8080

worker: Hi! 🤗
 We ❤ CLIs
but
wouldn’t it be convenient
to
query flows from Grafana?
Grafana datasource plugin

● Thank you Grafana for providing tutorials and libraries!

https://grafana.com/docs/grafana/latest/developers/plugins/
● Frontend: Typescript + React
● Backend: Go (Grafana Plugin SDK)
● Multiple underlying datastores
● Many visualizations
Learn more

Getting started with Hubble:

https://docs.cilium.io/en/v1.12/gettingstarted/#observability

Isovalent and Grafana announced partnership!

Blog post with use cases:
https://grafana.com/blog/2022/10/24/grafana-and-cilium-deep-ebpf-powered-obs
ervability-for-kubernetes-and-cloud-native-infrastructure/
Hands-on demo: https://github.com/isovalent/cilium-grafana-observability-demo
Thank you!