Networking Mesh

Mon, 01 Jan 0001 00:00:00 +0000

Kilo creates a WireGuard mesh between cluster locations. When running with Cilium, it uses IPIP encapsulation routed through Cilium’s VxLAN overlay so that traffic between locations works even when the cloud network blocks raw IPIP (protocol 4) packets.

Select the cilium-kilo networking variant

During platform setup, select the cilium-kilo networking variant. This deploys both Cilium and Kilo as an integrated stack with the required configuration:

How it works

Kilo runs in --local=false mode – it does not manage routes within a location (Cilium handles that)
Kilo creates a WireGuard tunnel (kilo0) between location leaders
Non-leader nodes in each location reach remote locations through IPIP encapsulation to their location leader, routed via Cilium’s VxLAN overlay
The leader decapsulates IPIP and forwards traffic through the WireGuard tunnel
Cilium’s enable-ipip-termination option creates the cilium_tunl interface (kernel’s tunl0 renamed) that Kilo uses for IPIP TX/RX – without it, the kernel detects TX recursion on the tunnel device

Talos machine config for cloud nodes

Cloud worker nodes must include Kilo annotations in their Talos machine config:

Local Cloud Controller Manager

Mon, 01 Jan 0001 00:00:00 +0000

The local-ccm package provides a lightweight cloud controller manager for self-managed clusters. It handles node IP detection and node lifecycle without requiring an external cloud provider.

What it does

External IP detection: Detects each node’s external IP via ip route get (default target: 8.8.8.8)
Node initialization: Removes the node.cloudprovider.kubernetes.io/uninitialized taint so pods can be scheduled
Node lifecycle controller (optional): Monitors NotReady nodes via ICMP ping and removes them after a configurable timeout

Install

cozypkg add cozystack.local-ccm

Talos machine config

All nodes in the cluster (including control plane) must have cloud-provider: external set so that kubelet defers node initialization to the cloud controller manager:

Multi-Location Clusters on Cozystack