nginx + keepalived VIP Failover: Parameter Matrix and 6 Production Incidents

Q: Why isn't my VIP floating after the primary goes down?

Three things to check in order: VRRP advertisements being sent (tcpdump -i eth0 vrrp), auth_pass matching byte-for-byte on both nodes (keepalived only reads 8 chars), and firewall rules allowing UDP 112 from the peer. Half the time it's the firewall.

Q: What's the correct value for virtual_routerpriority?

Lower number = higher priority. Primary: 100, Secondary: 90 is the classic setup. Don't use the same number on both — you'll get split-brain or flapping. If you need active-active, use a third node or a different protocol (LVS, BGP).

Q: Should I use nopreempt or preempt?

For most web backends: nopreempt. Once the secondary takes over, keep it until it dies. preempt causes VIP flapping whenever the primary recovers, which means DNS/cache churn and connection resets. Only use preempt if the primary has a specific role (stateful service, primary DB writer).

Q: Why is auth_pass in my config 32 chars but keepalived only uses 8?

Historical bug/feature — VRRP spec defines auth_pass as 8-char ASCII. keepalived silently truncates to 8 chars. If you set 32 chars, both nodes need to have the SAME first 8 chars. Common mistake: generate a 32-char random password on node A, paste on node B, and think it's secure. It's 8 chars.

Q: How do I health-check the upstream so keepalived actually fails over?

Use track_script or track_interface. For nginx: `track_script { script: check_nginx; interval: 5; fall: 3; rise: 2 }` and a bash script that curls localhost and returns non-zero on failure. Never trust a single check — require `fall: 3` before declaring failure and `rise: 2` before recovering.

Why keepalived, not BGP

If you have IPMI access, dynamic routing upstream, and a router willing to speak eBGP to a stateless VM, BGP is nicer. For two VMs behind a cloud VPC with a stateless load balancer in front, keepalived is the pragmatic answer: a small daemon speaking UDP 112 to the peer, moving the VIP in about three seconds. That's the whole pitch, and it's enough for most of us.

I've run keepalived in front of nginx at three companies and one non-profit. It works. It also fails in ways that eat on-call time if you don't know the failure modes ahead of time.

The happy path isn't the story

Every tutorial shows you the config, tells you to systemctl start keepalived, and shows the VIP floating when you kill the primary. In production you will not have that experience. You will have a firewall rule from a security audit three years ago, an auth_pass someone copy-pasted wrong, a NIC that got renamed after a kernel update, and a health check that only verifies the process is up rather than whether the service is serving. This article is about those cases.

Parameter matrix

Values for Ubuntu 22.04, keepalived 2.1.5, nginx 1.24.x.

Parameter	Safe value	What breaks if tuned wrong
`virtual_router_id`	`51` (identical both sides)	Different IDs = different VRRP groups = no failover, no errors, just silence
`virtual_routerpriority`	Primary `100`, Secondary `90`	Same on both = split-brain; higher on secondary = VIP on wrong box
`virtual_ipaddress`	`192.0.2.100/24` (identical)	Not matching = VIP moves but unreachable; wrong prefix = routes nowhere
`advert_int`	`1` (identical)	3x gap between peers = silent desync, occasional flapping
`garp_master_delay`	`5`	Too low = STP churn on the switch; too high = slow first packet after failover
`garp_master_repeat`	`2`	Reduces post-failover burst; clients with stale ARP keep hitting the dead primary
`garp_master_refresh`	`60`	Refreshes gratuitous ARPs; `0` disables and stale ARP wins
`garp_master_ttl`	Omit	Non-zero sets a lifetime; usually leave to the switch
`gna_interval`	`1`	Interval for Gratuitous ARP; too aggressive annoys neighbors, too slow leaves caches stale
`garp_validate_interface`	Omit	Validates the GARP NIC matches `interface`; only for weird NIC topologies
`nopreempt`	Yes, both sides	Without it, primary recovery yanks the VIP back — connection resets
`smtp_connect_timeout`	`3`	Long = delayed alerts, short = false alerts on slow SMTP
`script` (in `vrrp_script`)	Bash curl to localhost	Any static check (`pgrep nginx`) misses process-alive-but-not-serving
`auth_pass`	8-char ASCII, identical both sides	Only first 8 chars are used; longer strings are noise (see trap below)
`interface`	e.g. `eth0`	Wrong NIC = VIP moves to a NIC the client can't route to
`interface_broadcast`	Omit	Forces broadcast on a specific NIC; only needed on multi-homed setups
`virtual_router_track_interface`	e.g. `eth0`	If the interface dies, VRRP drops the VIP immediately rather than waiting
`notify_script`	Path to state-change hook	Fires on MASTER/BACKUP transitions; useful for logging and alerting

Case 1: VIP doesn't float when the primary dies

Symptom: kill nginx on the primary, wait 30 seconds, curl the VIP — still hitting the dead box.

Checklist, in order:

# 1. Are VRRP packets actually being sent?
sudo tcpdump -i eth0 -nn -c 5 vrrp

# 2. Do auth_pass values match byte-for-byte (first 8 chars)?
grep auth_pass /etc/keepalived/keepalived.conf | cut -c1-8

# 3. Is the firewall passing protocol 112?
sudo iptables -L INPUT -n | grep -Ei 'vrrp|112'

Nine out of ten: a firewall. VRRP is IP protocol 112, not a TCP/UDP port. Standard rules like -p tcp --dport 80 -j ACCEPT don't touch it, and a hardened host usually defaults to DROP which eats VRRP silently.

sudo iptables -I INPUT  -p vrrp -j ACCEPT
sudo iptables -I OUTPUT -p vrrp -j ACCEPT

Also check the cloud security group — many managed clouds let you filter on protocol number, and someone almost always forgets to add 112.

Case 2: Split-brain — both nodes hold the VIP

Symptom: after a primary reboot, both hosts show the VIP in ip addr. A client's ARP cache points at whichever box it hit first, so requests randomly work and randomly hang.

Detect:

arp -an | grep 192.0.2.100            # on a client
ip addr show eth0 | grep 192.0.2.100  # on both nodes

Root cause: priorities are equal (both think they're primary) or auth_pass mismatches (each side ignores the other and defaults to MASTER). Fix both:

vrrp_instance VI_1 {
    interface eth0
    virtual_router_id 51
    virtual_routerpriority 100   # 90 on the secondary
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass S3cret12
    }
    virtual_ipaddress {
        192.0.2.100/24
    }
}

Reload with systemctl reload keepalived — a reload, not a restart, is enough to clear the bad state on the losing node.

Case 3: Flapping every few minutes

Symptom: access logs show the VIP switching backends every 3–5 minutes, connections reset mid-flight.

I hit this the day I first deployed keepalived. The check was pgrep nginx. Under load, pgrep briefly returned a stale PID that was momentarily unresponsive; the check failed, VIP moved. Ten seconds later nginx recovered, check passed, VIP moved back. Every time.

Fix: nopreempt on both sides plus a slower, more deliberate check.

vrrp_script chk_nginx {
    script "/etc/keepalived/check_nginx.sh"
    interval 5
    fall 3
    rise 2
    weight -20
}

fall 3 at interval 5 means 15 seconds of consecutive failure before demotion. That's a real outage, not a hiccup.

Case 4: Wrong NIC on multi-homed hosts

Symptom: VIP moves to the secondary, ip addr shows it correctly, but curl 192.0.2.100 times out.

The interface field is where VRRP sends advertisements. On a host with two NICs — eth0 for business, eth1 for management — you want VRRP on the business NIC. Keepalived has three related knobs: interface (primary NIC for VRRP and VIP binding), interface_broadcast (forces broadcast on a specific NIC, useful on point-to-point links), and bind_ifid (assigns the virtual router ID to a specific interface so multiple VRs don't collide).

The mistake I made: interface eth0 on the primary, interface eth1 on the secondary. On failover, keepalived bound the VIP to eth1 — the mgmt network — and client traffic went nowhere.

Rule of thumb: same NIC name, same subnet, on both nodes. Add garp_master_delay 5 and garp_master_repeat 2 so the switch's MAC table updates without taking the port down.

Case 5: Health check reports up when the backend is down

This one costs the most money. Nginx is running, listening on 80, but the upstream app is dead and every request 502s. Keepalived, checking only nginx, says all clear.

Don't use systemctl is-active --quiet nginx. That's liveness, not readiness. Use a script that speaks HTTP to what the client would speak HTTP to:

#!/bin/bash
# /etc/keepalived/check_nginx.sh
set -e
curl -sf --max-time 3 http://127.0.0.1/healthz > /dev/null
curl -sf --max-time 3 http://127.0.0.1/upstream-probe > /dev/null

The /upstream-probe location returns 200 only when its upstream responds. Without it, nginx will happily 502 while still being "up". If you can't add an endpoint, at minimum curl the actual client URL. Set fall 3, rise 2 so a single blip doesn't trip the failover.

Case 6: Upgrade breaks config parsing

Upgrading keepalived from 2.0.x to 2.1.x is where I've lost the most time to parser errors. Two gotchas.

! comments in vrrp_script. Keepalived 2.0 used ! as a line comment inside script blocks. 2.1 removed it in some code paths. If your config has:

vrrp_script chk {
    script "/etc/keepalived/check.sh"  ! this comment is now an error
    interval 2
}

You'll see Error found in configuration file on reload, and systemd won't start keepalived. Convert all ! comments to #.

vrrp_version default flipped. Keepalived 1.4 defaulted to VRRPv2, 2.x to VRRPv3. Mixed-version peers don't interop. Pin it explicitly on both sides:

global_defs {
    vrrp_version 3
}

Always test with keepalived -t -f /etc/keepalived/keepalived.conf before systemctl reload. If the parser complains, the reload will silently keep the old config live while the new config fails to parse — the worst kind of half-state.

The auth_pass trap

VRRP's auth_pass is defined as 8 ASCII characters. Keepalived silently truncates anything longer. I've seen people generate a 32-char password with openssl rand -base64 24, paste it into both configs, and think they have strong VRRP auth. They have 8 chars.

Worse, if two nodes share the same string but differ in the first 8 characters (say S3cretPass1 vs S3cretPass2), the nodes don't reject each other's packets loudly — they just ignore the advertisements and each falls back to thinking it's primary. Split-brain via a typo, half an hour to find.

Test it: grep auth_pass /etc/keepalived/keepalived.conf | cut -c1-8 on both nodes, then diff.

Cheat sheet: 15 commands to bookmark

#	Command	What it tells you
1	`sudo tcpdump -i eth0 -nn vrrp`	Are VRRP advertisements flowing?
2	`sudo arping -I eth0 192.0.2.100`	Who currently owns the VIP's MAC?
3	`ip addr show eth0`	Is the VIP bound on this host?
4	`systemctl status keepalived`	Service state
5	`journalctl -u keepalived -n 200 --no-pager`	Recent state transitions
6	`keepalived -V`	Version — v2 vs v3 matters
7	`keepalived -t -f /etc/keepalived/keepalived.conf`	Config syntax check
8	`ss -tlnp \\| grep :80`	Is nginx listening?
9	`ss -tnp state established '( sport = :80 )'`	Established client connections on 80
10	`cat /etc/keepalived/keepalived.conf`	Actual loaded config, not what you remember writing
11	`tail -f /var/log/keepalived.log`	Old-style syslog output if you use it
12	`dig +short 192.0.2.100`	Reverse-resolve check
13	`ifconfig eth0`	Legacy alias for `ip addr`; some scripts still use it
14	`ip route get 192.0.2.100`	Which NIC the client kernel will use
15	`curl -v --max-time 3 http://192.0.2.100/`	End-to-end check through the VIP

FAQ

Why isn't my VIP floating after the primary goes down? Three things, in this order. Are VRRP advertisements actually being sent? tcpdump -i eth0 -nn vrrp on the surviving node — you should see packets from the dead primary's IP every second until it truly dies. Does auth_pass match byte-for-byte? Remember the 8-character truncation. Is the firewall allowing protocol 112? On iptables: iptables -I INPUT -p vrrp -j ACCEPT. Half the time I've debugged this, it's the firewall. The other half is auth_pass.

What's the correct value for virtual_routerpriority? Lower is higher priority. Classic setup: Primary 100, Secondary 90. The 10-point gap is enough for VRRP to decide who wins without tripping on the small weight adjustments the track scripts make. Never use the same priority on both nodes — you'll get split-brain or constant re-election. For true active-active, don't fight keepalived: VRRP is fundamentally active-passive at the VIP level. Use LVS or BGP with ECMP.

Should I use nopreempt or preempt? For nginx fronting a web app: nopreempt on both sides. Once the secondary takes over, let it keep the VIP until it dies. With preempt, every time the primary recovers from any blip it yanks the VIP back — connections reset, DNS caches churn. Only use preempt when the primary has a stateful role the secondary can't do and failing back is cheaper than manual promotion.

Why is auth_pass 32 chars in my config but keepalived only uses 8? VRRP defines auth_pass as an 8-byte string (RFC 3768). Keepalived reads the value, silently truncates to 8 ASCII bytes, and uses that. It never warns. So pasting a 32-char openssl rand -base64 string into both configs gives you 8 chars of auth — and both nodes just need to agree on those first 8. Two nodes with different full passwords but the same first 8 chars will happily interop; two nodes with different first 8 chars won't interop at all and each falls back to thinking it's primary — split-brain. Test with cut -c1-8 before deploying.

How do I health-check the upstream so keepalived actually fails over? Two layers. First, use vrrp_script with a curl-based check, not systemctl is-active — that only tells you the process is registered, not that it's serving. Second, tune fall and rise. I use interval 5, fall 3, rise 2 — 15 seconds of consecutive failures before declaring the node down, 10 seconds of consecutive successes before bringing it back. A single blip never trips the failover; a real outage does. Add weight -20 so a failing check loses the VRRP election to a healthy peer at priority 100.