cilium

VXLAN Encapsulation Series: Demystifying Cross-Node Pod Traffic in Kubernetes - Part 1

DRIDI Walid

— 4 min read
Share
VXLAN Encapsulation Series: Demystifying Cross-Node Pod Traffic in Kubernetes - Part 1
In Kubernetes environments, a very frustrating issue can happen: traffic between Pods on the same node works perfectly, but communication between Pods on different nodes hangs on large requests. Small packets like pings or basic TCP connections cross the nodes without any problem. However, larger transfers like TLS handshakes or big HTTP responses simply freeze and time out. This happens because the CNI overlay network adds extra header data, causing oversized inter-node packets to be dropped silently. When local Pod traffic works fine but cross-node traffic fails on large payloads, you are almost certainly facing an MTU (Maximum Transmission Unit) mismatch.

This issue almost always stems from a conflict between two key numbers: the MTU on the node and the MTU inside the pod. In most standard Kubernetes environments, the node interface uses an MTU of 1500, while the pod interface is set to 1450. That 50-byte difference is intentional—it accounts for the overlay header added to every cross-node packet.

However, public clouds work differently. On Google Kubernetes Engine (GKE), for instance, both nodes and pods use a 1460 MTU because GKE bypasses VXLAN overlay encapsulation by using direct VPC native routing with Cilium


The plan:

  1. See why there is a tunnel at all.
  2. Weigh the envelope, byte by byte.
Scope: Cilium with routingMode=tunnel and tunnelProtocol=vxlan, which is what you get when you install Cilium without choosing anything. 

The network has never heard of pod IPs

A Kubernetes cluster gives every node a slice of the pod CIDR. With the common 10.244.0.0/16, one worker gets 10.244.1.0/24 and the other 10.244.2.0/24. Those addresses exist only inside the nodes. Your switch, your router, your cloud VPC (Virtual Private Cloud) has no route for them. If a node put a packet for 10.244.2.187 on the wire as is, the first hop would drop it.

The Kubernetes network model makes this harder on purpose: pods must reach each other with their real IPs, no NAT (Network Address Translation) in between. So the CNI (Container Network Interface plugin) cannot simply masquerade pod traffic behind the node. Cilium's default answer is to wrap the pod packet inside a packet the network does understand. That is an overlay: a virtual pod network riding on top of the physical one.

  • VXLAN (Virtual Extensible LAN) puts the pod's whole Ethernet frame inside a UDP packet addressed from node to node. The network routes it like any UDP packet. The other node unwraps it and hands the original frame to the pod.
  • The thing that wraps and unwraps is the VTEP (VXLAN Tunnel Endpoint). In Cilium every node is a VTEP, through a kernel device named cilium_vxlan.
  • Each tunnel carries a 24-bit VNI (VXLAN Network Identifier). Cilium does something clever with it, which you will see on the wire.
  • Inside a node, a pod's eth0 is one end of a veth pair, a virtual cable between two network namespaces. The other end, named lxc*, sits in the node's namespace, and that is where Cilium attaches its eBPF programs: small programs the kernel verifies, then runs on every packet that crosses the interface.
Takeaway: the pod packet never changes. It travels inside an envelope, and the envelope costs 50 bytes.

The envelope, byte by byte

The envelope is four headers.

  • Two ways to count the same 50. On the wire, the frame grows by outer Ethernet 14 + IP 20 + UDP 8 + VXLAN 8. For the MTU (Maximum Transmission Unit, the largest IP packet an interface will send), the outer Ethernet header does not count, but the pod's own Ethernet header now travels as payload: IP 20 + UDP 8 + VXLAN 8 + inner Ethernet 14 = 50. Both sums give 50 because both Ethernet headers are 14 bytes. Cilium computes the second one: TunnelOverheadIPv4 = 50 in its source.
  • With an IPv6 underlay the envelope is 70 bytes and pods get 1430. A VLAN tag on the underlay changes nothing: it sits in the Ethernet header, outside the IP budget.
  • UDP 8472 is the Linux kernel’s legacy default VXLAN port (predating the IANA assignment of 4789), and Cilium uses it by default in VXLAN tunnel mode (configurable via tunnel-port; Geneve uses 6081, native routing uses no tunnel). If that port is blocked between nodes, pods on the same node can still talk, but cross-node pod traffic, including Service traffic to backends on other nodes, times out.
  • Subtract 50 from a 1500-byte underlay and pod-to-pod packets must stay at 1450.

What’s next?

In Part 2, we put theory into practice with a lab you can run in ten minutes: real captures of the envelope, the VNI decoded, and the 1450 checked on a live pod. Stay tuned!

Read more