Skip to content

nginx Upstream Load Balancing: Algorithm Selection, Server Directives, and the keepalive Trio

1. How nginx actually uses the upstream block

An upstream block is not just a list of servers — it's a peer pool that nginx evaluates on every proxied request. When a client hits proxy_pass http://api_pool, nginx does this work inside the worker process: resolve peer candidates (excluding down or over-max_conns), pick one via the configured algorithm, open or reuse a keepalive connection if available, proxy the request, and update that peer's failure counters if the response fails.

Every one of those steps is configurable. Most tutorials cover the third step and stop. The other four are where production incidents live.

2. Six algorithms, one decision matrix

Nginx ships with six upstream algorithms. The choice depends less on "which is fastest" and more on your traffic shape and backend state model.

Algorithm Config line Best for Kill it when
Round-robin (default) Identical pods, no session state Request durations vary wildly
Weighted round-robin server b weight=3 Mixed hardware or capacity Weights exceed real capacity ratio
ip_hash ip_hash at block level Legacy sessions without shared state Any nginx upgrade or peer add/remove
least_conn least_conn at block level Variable-duration requests (ML, reports) All peers roughly equal — adds bookkeeping cost
hash hash $cookie_sid consistent Cache-friendly routing (same key → same peer) Fewer than 2 peers per hash bucket
zone + consistent_hash zone pool 64m + consistent_hash Hot peer rotation during deploys Small peer counts (n < 4) — no benefit

The default round-robin is what I use for 70% of services. It is boring in the best way: predictable, tunable via weight, and no hidden bookkeeping. least_conn is where I land when P99 request duration is more than 5x P50 — otherwise the fairness gain isn't worth the extra counter churn.

3. Per-server directives, with real numbers

Every server line inside an upstream block accepts up to six parameters. Their actual impact varies by a factor of ten depending on traffic.

server 10.0.2.11:8080  weight=5   # Traffic share, 1..256
                     max_fails=3  # Failures within fail_timeout before pause
                     fail_timeout=10s  # Pause window; also failure window when >0
                     max_conns=200     # Hard cap on concurrent requests per peer
                     backup              # Only receives traffic when all primaries are down
                     down;               # Hard-remove this peer (use sparingly)

Values I use as defaults:

  • weight=1 for identical peers, weight=2 or weight=3 for pods with double or triple the CPU.
  • max_fails=3 with fail_timeout=10s for APIs. That means a peer needs three consecutive failures within ten seconds to be paused — one bad request does not kick it out, but a genuine outage does within seconds.
  • max_fails=1, fail_timeout=60s for expensive long-running jobs. One failure means pause for a full minute.
  • max_conns at 50% of the backend's own connection cap. For a Tomcat with maxThreads=200, I set max_conns 100. That headroom is deliberate — it lets nginx shed load before Tomcat's thread pool saturates.
  • backup on a canary peer with weight=1. Traffic only reaches it when every primary fails. This is my default for staged rollouts.

down deserves a warning: it removes the peer from the rotation without removing it from the file. It survives nginx -s reload. I keep a commented-out down line next to any server I might need to eject during an incident — flipping a # and reloading is faster than editing a list.

4. The keepalive trio that prevents 502 storms

The single most common nginx upstream misconfiguration I've debugged is a half-configured keepalive. The trio has to be complete:

upstream api_pool {
    server 10.0.2.11:8080 weight=5;
    server 10.0.2.12:8080 weight=5;
    server 10.0.2.13:8080 weight=5;

    keepalive 32;          # Max idle connections per worker to keep open
    keepalive_timeout 60s;  # Idle timeout on those sockets
    keepalive_requests 10000;  # Max requests per kept socket before recycling
}

server {
    location / {
        proxy_pass http://api_pool;
        proxy_http_version 1.1;            # Required — HTTP/1.0 cannot reuse sockets
        proxy_set_header Connection "";    # Strips "Connection: close" that browsers may send
    }
}

Get any one of these wrong and the effect is predictable:

  • Missing proxy_http_version 1.1; → nginx 502s with upstream sent no valid HTTP/1.0 header while reading response header from upstream.
  • Missing proxy_set_header Connection ""; → browser Connection: close propagates, keepalive never engages.
  • keepalive_requests too low (default is 1000) → sockets recycle every thousand requests, connect cost comes back.
  • keepalive_timeout shorter than backend's idle timeout → nginx closes first, backend gets a half-open and logs noise.

I use keepalive 32 because each nginx worker opens up to 32 idle sockets per peer pool. With four workers and three upstreams, that's 384 sockets maximum — well below typical ulimit -n. On heavy traffic I've pushed it to 64; below 16 you start missing reuse opportunities.

5. Health checks: the reality check

Nginx OSS has no built-in active health checking. When a peer fails a real request, nginx marks it down for fail_timeout and moves on. That is passive health checking, and it works — until the backend hangs without refusing connections. Then nginx keeps sending requests into a black hole.

Three ways around this:

  1. Passive + generous timeouts. Set proxy_connect_timeout 3s and proxy_read_timeout 30s. Anything slower trips max_fails within seconds. Simple, no code, but you're using client-visible latency as the health signal.
  2. balancer_by_lua_block with lua-resty-balancer doing its own periodic probes. Works, but you're now running Lua inside nginx and debugging two runtimes.
  3. nginx-plus with health_check directive. If you already pay for plus, use it. It does exactly what OSS lacks: periodic TCP/HTTP checks independent of traffic.

For OSS, I pair max_fails=3 fail_timeout=10s with an external health checker that touches the upstreams directly and alerts, rather than trying to encode health logic into nginx.

6. Zone directives and dynamic rebalancing

The zone directive is where upstream state moves out of per-worker memory into shared memory:

upstream api_pool {
    zone api_pool 64m;
    server 10.0.2.11:8080;
    server 10.0.2.12:8080;
}

Without zone, each worker maintains its own peer state. That means:

  • max_fails counters are per-worker, not global. A worker that just started won't know the other workers marked a peer down.
  • least_conn counts are per-worker. With 32 workers, you can have 32x more connections to one peer than to another.

With zone enabled, state is shared. The cost: nginx can no longer add or remove peers without a full reload. If you need runtime peer changes (say, an autoscaler adding pods), use lua-resty-balancer or OpenResty's resty balancing APIs instead.

Dynamic rebalancing through Lua costs you in three places: debugging (two languages), config drift (peers no longer live in the config file), and observability (nginx's built-in balancer status no longer reflects reality). I've done it once and I don't like it.

7. Five production incidents I've hit

7.1 Weighted round-robin starving a slow backend

Three pods, weights 5, 5, 1. The weight-1 pod kept getting 15% of traffic, which was more than its CPU could absorb. It fell behind, requests queued, then nginx marked it failed for fail_timeout — which kicked it off the rotation, which meant the other two got 100% load, which pushed them over their caps. The recovery I had to write: lower the healthy pods to weight=4 and the slow pod to weight=2, so the imbalance reflected actual capacity instead of aspirational allocation.

7.2 ip_hash breaking on nginx 1.25

Nginx 1.25 changed the internal hash seed for ip_hash. We upgraded a fleet of 40 nodes and roughly 30% of in-flight sessions silently moved to different backends. The app was fine with the wrong peer — it just lost session state, so users saw "please log in again." I downgraded, and we waited for a proper migration: a shared Redis session layer, then remove ip_hash entirely. The lesson is that hash-based affinity looks stable until it isn't.

7.3 Half-configured keepalive → 502 storm

I added keepalive 32 to an upstream block at 2am and forgot proxy_http_version 1.1;. Reload succeeded. Within 90 seconds every request 502'd. The error.log line was upstream sent no valid HTTP/1.0 header while reading response header from upstream. I've now got that line in my on-call runbook as a "keepalive miswired" trigger — it appears when you have keepalive configured but not correctly.

7.4 max_conns set too low → 503 under burst

Set max_conns=5 on a pool of eight pods. Peak traffic hit, one pod's counter tripped, nginx started returning 503 with no live upstreams while connecting to upstream. It looked like the entire pool was down. Lowering max_conns to 100 fixed it. My default now is: max_conns at 50% of backend capacity, never below 20 for anything that sees more than 50 rps.

7.5 Upstream timeout cascade from a single slow request

One backend took 45 seconds to respond to a legitimate request. proxy_read_timeout was 60s. The worker held the connection, the client waited, then the client disconnected at 30s — nginx logged a 499, the worker was now stuck on a socket the client no longer cared about. Multiply by one slow peer per second and you've exhausted worker_connections. Fix was proxy_read_timeout 15s on that endpoint, which forced nginx to time out the request and return 504 cleanly instead of holding a half-open socket.

8. Production template

Copy-pasteable, tuned for a generic API pool:

upstream api_pool {
    zone api_pool 64m;

    server 10.0.2.11:8080 weight=5 max_fails=3 fail_timeout=10s max_conns=200;
    server 10.0.2.12:8080 weight=5 max_fails=3 fail_timeout=10s max_conns=200;
    server 10.0.2.13:8080 weight=5 max_fails=3 fail_timeout=10s max_conns=200;
    server 10.0.2.99:8080 weight=1 max_fails=1 fail_timeout=60s max_conns=50 backup;

    keepalive 32;
    keepalive_timeout 60s;
    keepalive_requests 10000;
}

server {
    listen 443 ssl http2;
    server_name api.example.com;

    location / {
        proxy_pass http://api_pool;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Connection        "";

        proxy_connect_timeout 3s;
        proxy_send_timeout    30s;
        proxy_read_timeout    30s;

        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 5s;
    }
}

The three proxy_next_upstream lines are the piece most people forget. They tell nginx which response codes should trigger a retry on another peer, how many times, and within what time budget. Without them, a single peer outage looks like a service outage to the client.

9. Cheat Sheet — 15 upstream one-liners

# Show the upstream blocks actually loaded (not what you think you wrote)
nginx -T 2>&1 | awk '/upstream .* \{/,/\}/'

# Count current connections to each upstream peer
ss -tnp | grep ':8080' | awk '{print $4}' | sort | uniq -c | sort -rn

# See how many ESTABLISHED vs TIME_WAIT to the upstream pool
ss -tn state established '( dport = :8080 )' | wc -l
ss -tn state time-wait '( dport = :8080 )' | wc -l

# Test the keepalive trio is actually engaged
curl -s -o /dev/null -w '%{time_connect}\n' http://api.example.com/  # sub-millisecond = keepalive working

# Trigger a passive health check on a suspect peer
for i in $(seq 1 20); do curl -s -o /dev/null -w '%{http_code} %{time_total}\n' http://api.example.com/; done

# Drain a peer without killing it (mark down, reload)
sed -i 's|server 10.0.2.11:8080|server 10.0.2.11:8080 down|' /etc/nginx/conf.d/api.conf
nginx -s reload

# Bring it back
sed -i 's|server 10.0.2.11:8080 down|server 10.0.2.11:8080|' /etc/nginx/conf.d/api.conf
nginx -s reload

# Reload with zero connection loss
nginx -s reload && nginx -s reopen

# Check worker count and connection cap
nginx -T 2>&1 | grep -E 'worker_(processes|connections|rlimit_nofile)'

# Watch upstream errors as they happen
tail -F /var/log/nginx/error.log | grep -E 'upstream|connect\(\) failed|timed out'

# Look at nginx access log upstream timing
awk '{print $NF}' /var/log/nginx/access.log | head

# Show current upstream pool (nginx-plus only)
curl -s http://127.0.0.1:8081/api/upstream

# Compare peer load balance (nginx-plus)
curl -s http://127.0.0.1:8081/api/upstream/api_pool | jq '.peers[].received'

# Full config syntax + reload rehearsal
nginx -t && nginx -s reload

# Show the keepalive directives as loaded
nginx -T 2>&1 | grep -E 'keepalive|proxy_http_version|Connection'

10. FAQ

Which algorithm for stateless REST APIs behind 3 identical pods? Default round-robin. Don't overthink it — the algorithm cost is noise compared to backend CPU cost. Only reach for least_conn when request duration variance is high (P99/P50 > 5x). If pods differ in size, use weighted round-robin with weights proportional to vCPU.

Why do I get "upstream sent no valid HTTP/1.0 header" after adding keepalive? You set keepalive 32 inside the upstream block but forgot proxy_http_version 1.1; inside the location. Keepalive upstreams require HTTP/1.1 between nginx and the backend — HTTP/1.0, which is nginx's default, cannot carry connection reuse.

Does nginx have active health checks? No. OSS nginx does passive checks only — a peer is marked down after a real request fails. For real-time health checks you need nginx-plus, an external health checker updating the pool via the balancer API, or lua-resty-balancer doing its own probing.

What's a safe max_conns value? Start at 50% of the backend's own connection cap. Node.js with ulimit -n 65536 in a single process: max_conns 500. Java behind Tomcat with maxThreads=200: max_conns 100. Never go below 20 on anything receiving more than 50 rps — sub-20 values trip the "no live upstreams" fast path on bursts.

Should I use ip_hash for session affinity? Only if the backend holds sticky state and you cannot pass it through any other way. ip_hash is brittle: an nginx upgrade can rehash everything (this actually broke production in nginx 1.25), and the classic implementation does not fail over a dead peer. Prefer token-based session transfer, a shared Redis session layer, or $cookie/$header hashing when you control the keys.

How do I know if my keepalive trio is actually working? Watch three signals: TCP connection churn (ss -tnp | grep :8080 — mostly ESTABLISHED, not a fresh SYN every request), connect time (curl -w '%{time_connect}\n' — sub-millisecond on warm sockets, tens of microseconds to low milliseconds), and upstream bytes counters (steady rise, no sawtooth that tracks request volume). If connect time drops from ~30µs to ~2µs after adding keepalive, it's engaged.

How does zone change behavior? Without zone, each worker has its own peer state — max_fails counters and least_conn counts are per-worker, and peer add/remove requires a full reload. With zone, state is shared in memory across workers, and dynamic rebalancing becomes possible via the balancer API. The tradeoff: you can no longer edit peers freely in config without a reload, and debugging gets harder.