Node-local website

When using k8s as geo-distributed CDN, instead of how it was supposed to be used

Issue

By default, traffic in Kubernetes flows like this: first it reaches load balancer (LB), then gets forwarded to ingress controller (IC), and after that - to pod.

If you have LB running on one node¹, IC - on another, and pod - on a node different from where IC is running, then your traffic flows between these three nodes.

And if you have geo-distributed cluster, you might want to keep traffic local to each node. But if you simply run LBs and ICs on all nodes in your cluster - then LBs and ICs will choose downstream ICs and pods in round-robin fashion, meaning that sometimes traffic might end up in the pod on the same node as LB that received it, but getting there through IC running in another region! 🙀

Solution

To disable LB -> IC round-robin, you simply disable LB and expose IC directly on HTTP(s) nodePorts.

To disable IC -> pod (actually, service) round-robin, you first enable internalTrafficPolicy for the pod service, and then also add service-upstream annotation to your ingress.

Source

Thanks for this comment in k3s GitHub issue discussion. As author of that comment, I also blamed k3s first.