ARP-based "load balancing" on Kubernetes: one node does all the work!!!

DRIDI Walid

— 11 min read
Share
💡 kube-vip, MetalLB L2, Cilium L2 announcements, keepalived: these are different tools, but they all use the same trick. None of them give you load balancing. They give you a highly available VIP. (A VIP is a virtual IP address that can move between nodes.) That is not the same thing. You see the difference in four situations when things go wrong. Your traffic grows. You drain the node that announces the VIP (drain means to move all its pods away). Or you lose that node. Or an ARP cache keeps the old entry too long. (An ARP cache is a table that maps an IP to a hardware address.) And with kube-vip and Cilium, you see it when the API server is unreachable. Then the leader (the node that currently holds the VIP) gives up the VIP after a few seconds. No other node can take it.

Suppose you run Kubernetes on your own hardware. You need to give a Service an IP that people outside the cluster can reach. The easiest answer is layer 2 mode, or L2 mode. Layer 2 is the local network, where machines find each other by hardware address. Examples: MetalLB in L2 mode, kube-vip in ARP mode, or keepalived. You give the tool a pool of IPs from your LAN. It works in ten minutes.

Then one node ends up carrying everything. And you find out why.

How it actually works

First, four words. A MAC address is the hardware address of a network card. A NIC is a network interface card, the network port of a server. ARP is the protocol that maps an IP address to a MAC address on the local network. A VIP is a virtual IP that is not fixed to one machine. In this post the VIP is 10.0.0.50.

One node is elected leader for a given VIP. It answers ARP requests for that IP with its own MAC address. Now everyone on the LAN sends traffic for that IP to that one node. The other nodes stay quiet.

                    Router / Switch
                          |
        "Who has 10.0.0.50?"  -->  "I do, MAC aa:bb:cc"
                          |
       +------------------+------------------+
       |                  |                  |
   +---+---+          +---+---+          +---+---+
   |worker1|          |worker2|          |worker3|
   | LEADER|          | idle  |          | idle  |
   | VIP   |          |       |          |       |
   +---+---+          +-------+          +-------+
       |
   ALL traffic
       |
   +---v------------------------------------+
   |  kube-proxy / eBPF forwards to pods    |
   +---+-------------+-------------+--------+
       |             |             |
     pod A         pod B         pod C
   (worker1)     (worker2)     (worker3)

kube-proxy is the Kubernetes part that forwards Service traffic to pods. eBPF is a way to run small programs inside the Linux kernel. Cilium can use eBPF instead of kube-proxy. The pods are spread out. The entry point is not. That is the whole problem in one picture.

What goes wrong

One node's NIC is your upper limit

Every packet for that Service goes through one machine. If that node has a 10G card, your Service has 10G. It does not matter how many workers you add. The CPU on that node also does all the connection tracking. Connection tracking, or conntrack, is the Linux table that remembers every open connection. Add as many nodes as you like. The entry point stays the same size.

You get a little spreading if you expose several Services. But only when the tool elects a leader per VIP. MetalLB L2 does. It ranks the nodes with a hash per service IP. (A hash is a number computed from text. The same text always gives the same number.) Cilium L2 announcements do too. Each Service gets one Lease, first come first serve, so the spread is not balanced. (A Lease is a small Kubernetes object. The leader writes its name in it and renews it on a timer.) kube-vip does not, by default. Its setting svc_election (flag --servicesElection) is false by default. So a single leader holds every Service VIP. Set svc_election: "true" to get one election per Service. keepalived only spreads if you define one vrrp_instance per VIP. Each node then needs a different priority order. With one busy Service, you get nothing.

Failover is slow and drops connections

Failover means another node takes over the VIP. When the leader dies, another node has to notice. Then it takes the IP. Then it tells the network with a gratuitous ARP. A gratuitous ARP is an ARP message that nobody asked for. It says: "10.0.0.50 is now at this MAC address."

  t=0s       worker1 dies
             |
  t=0..D     nobody knows yet. Traffic goes into a black hole.
             D depends on the tool, not on ARP:
               keepalived   ~3 s     (three missed VRRP adverts)
               MetalLB L2   ~5 s     (memberlist gossip)
               kube-vip     ~15 s    (Lease, defaults 15/10/2)
               Cilium L2    10-20 s  (Lease, defaults 15/5/2)
             |
  t=D        worker2 takes the VIP, sends gratuitous ARP
             |
  t=D+ms     hosts that accept gratuitous ARP switch at once
             |
  t=D+???    hosts that ignore gratuitous ARP wait until their
             own ARP cache entry expires
             |
  t=?        traffic flows again for everyone

A black hole means traffic is sent but nothing answers. How long the black hole lasts depends on the tool. It depends on how the tool notices the dead leader. It does not depend on ARP.

  • keepalived uses VRRP. VRRP is a protocol where the master sends "I am alive" messages. After about three missed messages, the master is dead. That is about 3 s with the default advert_int of 1 s (3.6 s at priority 100). With an advert_int of less than one second (VRRPv3), it is under 1 s.
  • MetalLB L2 uses memberlist gossip. Gossip means the nodes ping each other directly, with no central server. It probes every 1 s with a 500 ms timeout. Then it waits about 4 s to be sure. The next leader is fixed by a hash, so there is no election. The IP usually moves within about 5 s. MetalLB's docs say to report a bug if failover takes more than about 10 s.
  • kube-vip and Cilium L2 hold a Kubernetes Lease. Nobody can take the VIP until the Lease expires. With kube-vip's defaults that is roughly 15 s (leaseDuration 15 s, renewDeadline 10 s, retryPeriod 2 s). With Cilium's defaults it is 10 to 20 s (15 s / 5 s / 2 s).

On top of that comes the ARP cache. Hosts and routers that accept gratuitous ARP switch at once. The tools differ in how often they repeat it:

  • MetalLB sends it at once, then every 1.1 s for about 5 s. A periodic --gratuitous-arp-interval exists, but it is off by default.
  • kube-vip re-sends it every 3 s (vip_arpRate, minimum 500 ms) for as long as it holds the VIP.
  • keepalived sends five at the MASTER change, and five more 5 s later (garp_master_repeat, garp_master_delay).
  • Cilium sends one gratuitous ARP reply per configured interface.

Clients that ignore gratuitous ARP do not wait a fixed 10 to 30 s. They wait until their own cache entry ages out. That is why a few clients take much longer to recover than the rest. Some switches ignore gratuitous ARP. Some clients keep a stale entry for minutes.

And nothing is synchronized between nodes. Every open TCP connection dies at the moment of the switch. The new leader does not know those connections existed. The same happens when you drain a node for a kernel upgrade, unless you plan the move.

Your VIP is stuck in one broadcast domain

A broadcast domain is the set of machines that receive each other's broadcast frames, usually one VLAN. ARP only works inside one broadcast domain. So the VIP has to live on the same L2 segment as your nodes. That kills a few designs:

  • Nodes in two racks with routing in between? The VIP can only live in one rack.
  • Two sites? Same problem.
  • Want per-rack subnets, like a good network engineer? You can't.

Expect resistance from the network team

Many people think L2 mode moves a MAC address around. For most of these tools, that is wrong. With MetalLB L2, kube-vip ARP mode and Cilium L2 announcements, the MAC never moves. The new leader answers ARP for the VIP with its own NIC's MAC. It sends gratuitous ARP. Only the IP-to-MAC binding in ARP caches changes. The switch's MAC table is untouched. Each node's MAC stays on its own switch port. keepalived is the exception when use_vmac is set. The VIP then lives on a macvlan interface. (A macvlan is a virtual interface with its own MAC.) It uses the VRRP virtual MAC 00:00:5e:00:01:{VRID} (00:00:5e:00:02:{VRID} for IPv6). That MAC really moves from one switch port to another on failover. In return, hosts' ARP caches stay valid.

Modern networks have features that check MAC addresses and ARP. Here is what each one does to you:

  • Port security is MAC-based. It counts and pins the source MACs seen on a switch port. It never looks at IP addresses or ARP contents. MetalLB, kube-vip and Cilium L2 only send with the node's own NIC MAC. So port security does not interfere with them. It does bite keepalived with use_vmac. After failover the virtual MAC appears on a different secure port. The switch treats that as a violation (err-disable in the default shutdown mode). keepalived without use_vmac uses the node MAC and is unaffected.
  • Dynamic ARP Inspection (DAI) checks each ARP message against a trusted IP-to-MAC list. It can drop your gratuitous ARP as a spoofing attempt. IP Source Guard does the same check on IP packets. Both hit all four tools.
  • MAC-flapping alarms do not fire on failover with MetalLB L2, kube-vip ARP, Cilium L2, or keepalived without use_vmac. No MAC changes switch port. Only keepalived with use_vmac moves the virtual MAC between ports. A clean failover is a single MAC move, at most one "MAC moved" log line. Repeated flap alarms (Cisco %SW_MATM-4-MACFLAP_NOTIF) mean split-brain. Split-brain means two nodes both act as master and both send frames from the virtual MAC.
  • vSphere: the vSwitch and VDS policies "MAC address changes" and "Forged transmits" are set to Reject by default. This has been the default since vSphere 7.0. They do not affect MetalLB L2, kube-vip ARP or Cilium L2. Every ARP reply and data frame still leaves with the vNIC's own MAC. They do break keepalived with use_vmac. Frames from the virtual MAC are dropped as forged transmits. Inbound frames to that MAC are not delivered without promiscuous mode or VDS MAC learning. Any macvlan or bridge setup with a different MAC breaks too. If MetalLB L2 fails on vSphere, look at NSX SpoofGuard or a port-group / VLAN mismatch. NSX SpoofGuard checks the IP-to-MAC binding, and that hits all four tools.
  • OpenStack: port security checks the IP-to-MAC binding and drops the traffic. You must add the VIP as an allowed address pair. This hits all four tools.

Most "MetalLB doesn't work" tickets I've seen come down to one of these.

The leader might not even host a pod

externalTrafficPolicy is a Service setting with two values. Cluster: the node may forward to a pod on any node. Local: the node only sends to pods on itself.

With externalTrafficPolicy: Cluster, the leader forwards to a pod on another node. That adds a hop. It also hides the client IP behind SNAT. SNAT is source NAT: the node replaces the client IP with its own IP. Your logs show node IPs instead of real users.

client --> worker1 (VIP, no pod) --> SNAT --> worker3 (pod)
                                     ^
                              client IP lost here

With Local, you keep the client IP. But the node holding the VIP must have a ready local pod. If not, traffic is silently dropped. The tools differ in whether they enforce that. MetalLB L2 does: nodes without ready endpoints are excluded from the announcer candidates. kube-vip does only if svc_election is set to "true" (default false). That setting restricts the election to nodes with a local pod. Cilium's L2 announcement docs call externalTrafficPolicy: Local incompatible with the feature. The VIP can be announced from a node with no pod. keepalived knows nothing about pods. You have to wire endpoint health into VRRP track_script / notify scripts yourself. And with Local, a pod restart can trigger a VIP move. That means another ARP cycle and another round of dropped connections.

Debugging gets annoying

Ask "which node serves this IP right now?" and you have to go look. Traffic graphs per node look wrong. One node is at 90% and the rest are flat. New people on the team think the cluster is broken.

How the leader is chosen: Lease, gossip, or VRRP

All four tools must pick one node per VIP. They do it in three ways.

1. A Lease in the Kubernetes API (kube-vip, Cilium). The leader writes its name in a Lease object and renews it on a timer. The other nodes watch the Lease. If it is not renewed for LeaseDuration, they try to take it. The first write that reaches the API server wins. Good points: it reuses the node's existing connection to the API server. No extra port or protocol between nodes. Access is controlled by normal Kubernetes RBAC (permission rules). There is one arbiter, the API server, so two leaders at once are rare. Bad points: the data plane (the path that carries user traffic) now depends on the API server. If the API server is lost for longer than the renew deadline, the leader gives up. Failover is bounded by LeaseDuration + RenewDeadline. Faster timers cost API traffic. Cilium's rule: QPS = #services × (1 / leaseRenewDeadline). Example: 65 Services at 2 s = 32.5 QPS. And nobody fences the old leader.

2. Gossip between nodes with memberlist (MetalLB). memberlist is a library that implements the SWIM gossip protocol. Every speaker pings the others on port 7946, TCP and UDP. The messages are encrypted with a shared key from a Kubernetes Secret named memberlist. There is no Lease and no election round. Each speaker sorts the nodes by a hash of "node + VIP". The first node in the list announces. A dead node is confirmed dead about 5 s after the first failed probe. Good points: the VIP's liveness does not depend on the control plane. Failover on node death needs no API call. Bad points: port 7946 must be open between nodes. During a network partition, two speakers can both think they are first. Then both answer ARP for the same IP (MetalLB issue #2838, still open). memberlist itself says partitions are only "partially tolerated". And when a node returns, the VIP flaps back to it if it ranks first.

3. VRRP (keepalived). VRRP is RFC 5798. It runs directly on IP protocol 112, to multicast address 224.0.0.18. The master sends an advert every 1 s by default. The highest priority wins (1 to 254, default 100). A backup declares the master dead after 3 adverts plus a skew time. That is about 3.6 s at priority 100. By default a returning higher-priority node takes the VIP back. The nopreempt keyword stops that. Good points: no Kubernetes dependency at all. The kubeadm HA guide calls it "well-known and well-tested". Sub-second failover with a fractional advert_int. Bad points: it knows nothing about Services, pods or drains. You write the health checks (vrrp_script, track_script) and notify scripts yourself. All nodes must be in the same IP subnet. Filtered adverts cause split-brain. There is no BGP mode; pair it with FRR or BIRD.

Cilium splits the work in two parts. LB IPAM gives IPs to LoadBalancer Services from a CiliumLoadBalancerIPPool. L2 announcements answers ARP for those IPs.

My verdict. Gossip and VRRP keep the VIP alive without the control plane. The VIP survives an API server outage. It moves on node death with no API call. The price is one more protocol between nodes, and a small risk of two announcers during a partition. Lease-based tools are simpler to run. No extra ports, RBAC-governed, one arbiter. But they tie the data plane to the API server. Lose it longer than the renew deadline and a healthy node drops the VIP.

So this is what I pick:

  • Application Services on-prem or in a homelab: MetalLB L2. It needs one open port (7946) and no CNI change. (The CNI is the cluster's network plugin.)
  • Cilium L2 announcements: only if you already run Cilium with kube-proxy replacement. Accept the 10-20 s failover and the API server coupling.
  • The kubeadm control-plane VIP: kube-vip as a static pod. It can also serve Services. Know that it depends on the API server it fronts, through the Lease. Since v1.29 you need the super-admin.conf workaround (issue #684).
  • A VIP with no Kubernetes dependency: keepalived + haproxy. The kubeadm HA guide lists it first as "well-known and well-tested". It is also the tool for an HA pair of external load balancers.
  • When the network team can set up BGP peering with your nodes: BGP. (BGP is the routing protocol that routers use to share paths.) Every node advertises the VIP, and ECMP (equal-cost multi-path routing) spreads the flows. Among the tools in the table, only MetalLB's FRR-based modes offer BFD today. kube-router has it too, via GoBGP. BFD is a fast "are you alive?" check between router and node. Cilium's BGP Control Plane does not have it. Lower holdTimeSeconds (min 3) and keepAliveTimeSeconds (min 1) instead.
💡 L2 mode gives you an IP that survives a node failure. It does not give you load balancing. It caps your throughput at one NIC. It drops connections on failover. It locks your VIPs into a single broadcast domain. And with kube-vip and Cilium, it needs the API server to stay alive.

If you're small, use it and move on. If you're growing, talk to your network team about BGP. Do it before your traffic grows so much that you have no other choice.