Cilium Cluster Mesh:

The EBPF-Powered Multicluster Solution for

Kubernetes

Abdul Basit
Product Architect
Rakuten Symphony SG
Symcloud Offers 3 Products
1 Symcloud Platform
Cloud Native Platform
Fully-integrated enhanced-
Kubernetes platform, optimized
for running Storage- and
Network- intensive services with
special focus on zero-touch
deployment and industry leading
operational automation at the
Edge
2 Symcloud Storage
Cloud Native Storage
Highest performing cloud-native
storage stack for Kubernetes with
special focus on storage and
application-aware data
management, where volumes +
applications can be manipulated as a
whole, as one single logical group.
Symcloud Orchestrator
3 Baremetal and Service
Automation
Highly scalable Infrastructure
and Service/Application
orchestrator to manage
baremetal servers, and
Applications across 100K+
servers and 10K+ clusters,
datacenters
2
 What is Cilium
Cilium is an open-source CNCF graduated project that provides
• Networking : CNI, LB, ClusterMesh, Bandwidth Manager,
BGP, Egress Gateway, KPR
• Security: Transparent Encryption, Network Policy, Run Time
Enforcement
• Observability: Metrics, Tracing, Network Flow Logs, Service
Maps
• ServiceMesh: Gateway API, Traffic Management, SPIFEE
At the foundation of Cilium is a new Linux kernel technology
called eBPF, which enables the dynamic insertion of powerful
security, visibility, and networking control logic into the Linux
kernel.

3
4
What is eBPF

Event driven kernel hook points that makes the

kernel programable in a secure and efficient way.
“What JavaScript is to the browser, eBPF is to the
linux kernel”

5
 Ingress-based cross cluster connect

• Simple to setup
• Independent Clusters with Network Fabric
no relationships between
workloads Cross-Cluster

• No k8s-native cross-cluster
service-discovery source

• No cross-cluster failover,
load-balancing, security
• Usually, the starting point to APP 1 APP 2 APP n APP 1 APP 2 destination
move to the proper multi-
cluster setup
Cluster1 Cluster2

6
 Service mesh-based multi-cluster

• Creating a Single logical cluster

of Multiple clusters
• Multicluster service-discovery Network Fabric

• Multicluster network routing

• Provides Global endpoints for
observability, routing and
source source
security destination
APP 1
• Support for complex scenarios
like federated identity, global
routing, and multi-cloud Global endpoints
deployments
Cluster1 Cluster2
Logical-Cluster

7
 Service Mesh Multicluster Use Cases
Availability
• Global Services across different kubernetes
clusters and even different cloud providers
• Better Utilization and failovers
• Locality Aware routing
Security
• Global identities for enforcing Security
policies across multiple clusters
Observability
• Global observability across clusters using
Hubble

Ref:https://isovalent.com/blog/post/topology-aware-routing-and-service-mesh-across-clusters-with-cluster-mesh 8
Service Mesh multi-cluster requirements

cross-clusters network
• Cross-cluster service discovery
source destination
• Network connectivity and load-balancing Networking

Important Add-ons
kube-api
• Monitoring and observability
• Encryption kube-api
endpoints discovery

• Access Control Cluster1 Cluster2

• Advance Traffic Management

9
Cilium Cluster Mesh Overview

• Service Mesh
• Service Discovery and Load
balancing
• Identity Aware Security Policies
• Observability
• Encryption
• Networking

Ref:https://isovalent.com/blog/post/topology-aware-routing-and-service-mesh-across-clusters-with-cluster-mesh 10
Cluster Mesh Architecture

Ref:https://isovalent.com/blog/post/topology-aware-routing-and-service-mesh-across-clusters-with-cluster-mesh 11
Cluster Mesh Requirements
• Same datapath modes for all clusters
• All Nodes must be reachable
• PodCIDR in all clusters must be non-conflicting
• Open required firewall ports for inter-cluster traffic
• Unique cluster name
Additional Requirements for Native-routed Datapath Modes:
• Pods in all clusters must have IP connectivity between each other.

12
Demo – Two clusters, Unique CIDRS

2001:db8:42:0::/56 2001:db8:43:0::/56

Agent Agent

KIND Cluster1 KIND Cluster2

13
Demo – BGP for Native Routing

2001:db8:42:0::/56 – cluster1
IPv6 Native-Routing using BGP
2001:db8:43:0::/56 – cluster2

2001:db8:42:0::/56 2001:db8:43:0::/56
Agent Agent

KIND Cluster1 KIND Cluster2

14
Demo – Setup ClusterMesh

2001:db8:42:0::/56 – cluster1
IPv6 Native-Routing using BGP
2001:db8:43:0::/56 – cluster2

2001:db8:42:0::/56 2001:db8:43:0::/56

Cluster
Agent Agent
Cluster
Mesh API Mesh API
Server Server

KIND Cluster1 KIND Cluster2

MetalLB MetalLB

15
Demo – Cross Cluster Traffic (EW)

2001:db8:42:0::/56 – cluster1
IPv6 Native-Routing using BGP
2001:db8:43:0::/56 – cluster2

sleep

Helloworld-v1 Helloworld-v2

2001:db8:42:0::/56 2001:db8:43:0::/56

Cluster
Agent Agent
Cluster
Mesh API Mesh API
Server Server

KIND Cluster1 KIND Cluster2

MetalLB MetalLB

https://github.com/abasitt/kube6/tree/main/cilium/kubedaySG2023 16
Demo – External Traffic (NS)

2001:db8:42:0::/56 – cluster1
IPv6 Native-Routing using BGP
2001:db8:43:0::/56 – cluster2
Istio-Ingress

sleep

Helloworld-v1 Helloworld-v2

2001:db8:42:0::/56 2001:db8:43:0::/56

Cluster
Agent Agent
Cluster
Mesh API Mesh API
Server Server

KIND Cluster1 KIND Cluster2

MetalLB MetalLB

https://github.com/abasitt/kube6/tree/main/cilium/kubedaySG2023 17
Thank you

18