Nginx DNS resolution for backend services

Running NGINX in Kubernetes reveals a subtle DNS resolution quirk

Kubernetes provides DNS for connecting from pods to services. Services are exposed as stable DNS names, but the IP addresses behind them can and do change as pods¹ are rescheduled, scaled, or restarted. Also, running helmfile destroy; helmfile sync recreates services, changing their IPs but not DNS names.

However nginx usually does not behave in a dynamic way when it comes to DNS. When a backend hostname is configured directly in proxy_pass or in an upstream block, nginx resolves that hostname once at startup and then caches the result indefinitely. If the underlying service IP changes, nginx keeps sending traffic to the old address, leading to failed requests.

To change nginx to dynamic DNS resolution, the backend address must have a variable in it. When proxy_pass contains a variable, nginx performs DNS resolution at each request. Probably this is somehow documented at nginx docs website, proxy_pass directive.

Unfortunately, in that case nginx request rewriting stops working.

For example, without vars:

location /name/ {
    proxy_pass http://service/remote/;
}

request /name/alex will be proxied to backend as http://service/remote/alex.

However, with variables:

set $backend "service";
location /name/ {
    proxy_pass http://$backend/remote/;
}

request /name/alex will be proxied to backend as http://service/remote/, losing actual name from original request. To fix it, you have to (re)construct desired URL manually, using nginx vars, like this:

location /name/ {
    proxy_pass http://service/remote/$request_uri$is_args$query_string;
}

You can, naturally, keep only those vars that you need, and add those you want.

That's all!