iptables Port Forwarding in Production — SNAT/DNAT Chain Architecture, Failure Modes, and nftables Migration
I've been on-call for a home-office NAT box and a small regional POP that each forward a handful of public ports into private VLANs. That sounds simple until 3 a.m. someone's Slack channel goes red because port 8443 stopped accepting handshakes.
This article is a distillation of those nights. I'm going to walk you through the actual chain of rules that makes port forwarding work, why each rule is non-optional, the five failure modes I've personally paged on, and a clean path from iptables to nftables on modern Ubuntu.
1. The Four-Rule Minimum
The whole mechanic of port forwarding is: rewrite the destination before the packet hits the router's local tables, let the FORWARD chain pass it, keep the state, then rewrite the source so the return trip comes back through you.
# (1) Rewrite the destination on the way in
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8080 \
-j DNAT --to-destination 192.168.1.10:80
# (2) Allow the forwarded packet through
iptables -A FORWARD -i eth0 -o eth1 -d 192.168.1.10 \
-p tcp --dport 80 -j ACCEPT
# (3) Allow the return path without re-inspecting it
iptables -A FORWARD -i eth1 -o eth0 -s 192.168.1.10 \
-p tcp --sport 80 -m state --state ESTABLISHED,RELATED -j ACCEPT
# (4) Rewrite the source so the originator sees your IP
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.10 \
-j SNAT --to-source 203.0.113.10
Remove any single rule and one leg of the trip silently dies. I have removed rule (3) a hundred times by accident while cleaning up a filter table. The symptom is always the same: SYN goes out, SYN-ACK comes back, gets dropped, client retries, times out. You never see an RST because you're the one killing it.
Rule (1) fires in PREROUTING before routing, so the destination you rewrite to doesn't need to be routable on the local interface — it just needs to be reachable from eth1. That's the whole point of DNAT.
2. conntrack State Is Not Optional
The -m state --state ESTABLISHED,RELATED -j ACCEPT rule on line 3 is what stops you from turning your NAT box into a stateless dumb repeater. Without it, you have to enumerate every return flow manually, which breaks the moment a client sends a mid-stream ICMP or does a bidirectional TCP push.
conntrack keeps one entry per connection (source IP, source port, dest IP, dest port, protocol, plus optional L4 marks). On my NAT box a busy afternoon pushes the table to around 40 000 entries:
$ cat /proc/sys/net/netfilter/nf_conntrack_count
38214
$ cat /proc/sys/net/netfilter/nf_conntrack_max
131072
Two knobs matter. nf_conntrack_tcp_timeout_established defaults to 432 000 seconds (5 days). If your upstream load balancer closes sessions in 15 minutes, you're holding onto slots you don't need. I set it to 900 on boxes doing a lot of short-lived TLS:
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=900
sysctl -w net.netfilter.nf_conntrack_max=262144
Persist those in /etc/sysctl.d/99-nat.conf or they'll reset on reboot and you'll spend another Tuesday wondering why your conntrack grew.
3. Real Topology — One NAT Box, Two LAN Segments
My home lab NAT box is a mini-PC with two interfaces. External is on 10.0.0.1/24 behind a residential router (public IP assigned via CGNAT). Internal is a dedicated switch on 192.168.1.0/24 with three real machines and a Docker host.
Internet NAT box Internal LAN
───────────┐ eth0 ┌──────────────┐ eth1 ┌─────────────────┐
203.0.113. | ───────▶ | 10.0.0.1 | │ 192.168.1.10 :80 │
.10 | | 192.168.1.1 | ────────▶│ 192.168.1.20 :443│
└──────────| | │ 192.168.1.30 :5432│
└──────────────┘ └─────────────────┘
Full rule set for this box:
# Enable forwarding
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.conf.all.rp_filter=2
# DNAT the three public ports
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.10:80
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to-destination 192.168.1.20:443
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 5432 -j DNAT --to-destination 192.168.1.30:5432
# Forward chain (both directions, stateful)
iptables -A FORWARD -i eth0 -o eth1 -m state --state NEW -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
# SNAT for outbound from internal
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j MASQUERADE
rp_filter=2 is the loose mode. I learned about that one the hard way in section 7.
4. Failure Mode 1 — Stateful Firewall Reset on Kernel Upgrade
Last month I debugged a case where a Friday-night apt full-upgrade rolled my box from Ubuntu 22.04.4 to 22.04.5 and every forwarded port went dark. The NAT rules survived because I had iptables-persistent installed. But my FORWARD rules did not.
Turns out iptables-persistent on 22.04 saves to /etc/iptables/rules.v4, which is loaded at boot by netfilter-persistent. Kernel upgrades don't trigger a reboot automatically, but unattended-upgrades did, and something in the boot chain raced with networkd — the rules loaded before the interfaces were up, so the -i eth0 matcher bound to nothing.
The fix isn't subtle:
[Service]
ExecStart=
ExecStart=/sbin/iptables-restore < /etc/iptables/rules.v4
Requires=network-online.target
After=network-online.target
Before I did that, my forward rules silently existed in memory with zero bindings and iptables -L showed them as if they were fine. Packet captures told the truth.
5. Failure Mode 2 — conntrack Table Exhaustion
In June 2025 a marketing event spiked connections to a public TLS endpoint on my NAT box. New-connection rate hit about 4 000/s for 90 seconds. The conntrack table filled:
$ cat /proc/sys/net/netfilter/nf_conntrack_count
65536
$ cat /proc/sys/net/netfilter/nf_conntrack_max
65536
$ dmesg | tail -30
[ 4123.881] nf_conntrack: table full, dropping packet
Default nf_conntrack_max on stock Ubuntu 22.04 is 65 536. Any host doing NAT for more than a few dozen clients will hit this. My fix:
cat >> /etc/sysctl.d/90-conntrack.conf <<EOF
net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_tcp_timeout_established = 900
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_tcp_be_liberal = 1
EOF
sysctl --system
I also installed conntrack-tools so I can query live:
The lesson: size nf_conntrack_max against your peak new-connection rate, not your peak concurrent connections. They're different numbers and ops teams usually guess at the wrong one.
6. Failure Mode 3 — MASQUERADE on a Static IP Silently Slows Everything Down
I set up a fresh NAT box with a static public IP assigned by my ISP. Standard muscle memory had me writing MASQUERADE on POSTROUTING. Traffic worked. For six months it worked fine.
Then I profiled a bursty workloads and noticed p99 latency was 40 % higher than expected. tcpdump on both sides told me the answer:
# With MASQUERADE — every SYN has a re-resolution overhead
tcpdump -i eth0 -n 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' \
-tttttt
10:23:41.123456 IP 192.168.1.10.4412 > 93.184.216.34.80: Flags [S], seq 1
10:23:41.124901 IP 203.0.113.10.51822 > 93.184.216.34.80: Flags [S], seq 1
10:23:41.125012 IP 192.168.1.20.39002 > 93.184.216.34.80: Flags [S], seq 1
10:23:41.126437 IP 203.0.113.10.51823 > 93.184.216.34.80: Flags [S], seq 1
Notice the 1.4 ms gap between each pair. With SNAT the gap is under 10 microseconds. The rule engine re-reads the source interface's current IP on every match because MASQUERADE is designed for DHCP-assigned egress. On a static IP there's nothing to re-resolve — I just wasn't telling the kernel that.
# Before (slow path)
iptables -t nat -D POSTROUTING -o eth0 -s 192.168.1.0/24 -j MASQUERADE
# After (fast path)
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j SNAT --to-source 203.0.113.10
Only reach for MASQUERADE when the egress IP actually changes.
7. Failure Mode 4 — NAT Bypass via a Second Gateway
This one bit me on a POP box that had two uplinks: a primary WAN on eth0 and a backup WAN on eth1 with a static default route. Both went to the same upstream provider. I wrote SNAT only for eth0. Traffic from the internal VLAN sometimes took eth1, skipped NAT entirely, and appeared on the internet with the source IP 192.168.1.10 — which is unreachable everywhere.
$ ip route show
default via 10.0.0.1 dev eth0 proto static
default via 172.16.0.1 dev eth1 proto static metric 20
192.168.1.0/24 dev eth1 scope link ← same interface as backup WAN, oops
Two problems, one box. First, two default routes means Linux ECMPs them and half my internal traffic was never going through NAT. Second, rp_filter=1 (the strict default) was silently dropping return packets on eth0 because they arrived from an interface that didn't match the FIB lookup for 192.168.1.10.
Fix:
# Delete the duplicate default
ip route del default via 172.16.0.1 dev eth1
# Persist via netplan
ip netns exec ... ip route replace 172.16.0.0/24 via 172.16.0.1 dev eth1 scope link
# Relax rp_filter so DNAT return packets are accepted on either interface
sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl -w net.ipv4.conf.eth0.rp_filter=2
sysctl -w net.ipv4.conf.eth1.rp_filter=2
If you do need two uplinks for redundancy, do it in policy routing with named tables, not with two default entries on the main table.
8. Failure Mode 5 — iptables-nft Coexistence Pitfalls
Ubuntu 22.04 ships iptables-nft as the default /usr/sbin/iptables symlink. All my muscle-memory iptables commands execute fine — they just get converted into nft rulesets under the hood. That's fine until you start mixing syntaxes.
Three things to remember:
The -w lock timeout. nft wraps a global lock. If a systemd-networkd restart is in flight when your script tries to install rules, you'll see:
iptables: Unable to open /run/xtables.lock. Another application holds it?
iptables: Waiting for existing lock ... 5.049 seconds passed
Always pass -w 30 when scripting. It waits up to 30 seconds for the lock instead of failing fast.
The legacy chain-name trap. iptables-legacy understands PREROUTING, OUTPUT, POSTROUTING, INPUT, FORWARD, and RAW as-is. nft translates them into nat prerouting, filter output, etc. If your script writes -A PREROUTING without -t nat, iptables-nft silently puts it in the filter table and it does nothing. The iptables-legacy binary catches this; nft does not.
The symlink mess. On a machine that had both packages installed the symlink could flip:
I switched back to iptables-legacy once because iptables-nft refused to load a ruleset that used the raw table with PREROUTING priorities that legacy accepted. Don't go there unless you have to — you're fighting the compat layer.
9. Migrating to nftables (Ubuntu 22.04+)
On Ubuntu 22.04 you probably don't need to migrate at all — you're already on nftables. But if you're on CentOS 7, Debian 10, or an older Ubuntu, here's the safe path:
# 1. Dump current rules
sudo iptables-save > /root/rules.v4.bak
sudo ip6tables-save > /root/rules.v6.bak
# 2. Install nftables
sudo apt update
sudo apt install -y nftables
# 3. Convert iptables ruleset to nft syntax
sudo iptables-save | sudo /usr/share/nftables/iptables-nft.sh | sudo nft -f -
sudo ip6tables-save | sudo /usr/share/nftables/iptables-nft.sh | sudo nft -f -
# 4. Verify
sudo nft list ruleset | less
sudo nft list table ip nat
sudo nft list ruleset | grep -E 'dnat|snat'
# 5. Remove the legacy shim AFTER verifying traffic
sudo apt remove -y iptables iptables-persistent
sudo apt install -y nftables
The iptables-nft.sh script is a translator, not a perfect conversion. Hand-fix things like owner matches, hashlimit, and complex -m multiport chains. Test each ruleset with nft -c -f - (check mode) before applying:
To watch live events during cutover:
sudo nft monitor trace
sudo nft monitor add # only rule additions
sudo nft monitor delete # only deletions
For my topology in section 3, the equivalent native nftables rule set is:
#!/usr/sbin/nft -f
table ip nat {
chain prerouting {
type nat hook prerouting priority dstnat; policy accept;
iifname eth0 ip daddr 203.0.113.10 tcp dport 80 dnat to 192.168.1.10:80
iifname eth0 ip daddr 203.0.113.10 tcp dport 443 dnat to 192.168.1.20:443
iifname eth0 ip daddr 203.0.113.10 tcp dport 5432 dnat to 192.168.1.30:5432
}
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
oifname eth0 ip saddr 192.168.1.0/24 snat to 203.0.113.10
}
}
table ip filter {
chain forward {
type filter hook forward priority 0; policy drop;
iifname eth0 oifname eth1 ct state new,untracked accept
iifname eth1 oifname eth0 ct state established,related accept
}
}
Notice ct state instead of -m state — nftables folded the state match into the ct meta key, and the ! negation is a prefix, not a flag.
10. 15-Command Verification Checklist
Run these in order after any rule change. Stop as soon as one fails.
# 1. Kernel engine
iptables -V
ls -l /usr/sbin/iptables
# 2. Sysctls
sysctl net.ipv4.ip_forward
sysctl net.ipv4.conf.all.rp_filter
# 3. NAT rules
iptables -t nat -L -n -v --line-numbers
# 4. Filter rules
iptables -L -n -v --line-numbers
# 5. Match counts (look for zeros)
iptables -t nat -L PREROUTING -n -v
iptables -t nat -L POSTROUTING -n -v
# 6. Conntrack health
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
dmesg | grep -i conntrack | tail -5
# 7. Routing
ip route show table all
# 8. Listening sockets on internal host
ssh 192.168.1.10 'ss -lntp | grep :80 '
# 9. End-to-end from internet
curl -vI --resolve test.example:8080:203.0.113.10 http://test.example:8080/
# 10. Return path
tcpdump -i eth1 -n 'host 192.168.1.10 and port 80' -c 5
# 11. Forward direction
tcpdump -i eth0 -n 'host 203.0.113.10 and port 8080' -c 5
# 12. Trace
sudo nft monitor trace 2>&1 | head -40
# 13. Persistence
ls -l /etc/iptables/rules.v4
systemctl status netfilter-persistent --no-pager
# 14. Lock contention
ls -l /run/xtables.lock
# 15. Rule count sanity
iptables -S | wc -l
nft list ruleset 2>/dev/null | wc -l
If command 6 shows table full in dmesg, jump to section 5. If command 13 has an empty rules.v4, your rules will be gone at next reboot. If command 8 shows nothing listening, no amount of iptables is going to help.
Keep this file on the NAT box itself as /usr/local/bin/portfwd-check.sh. You'll wish you did the first time a ticket shows up at 3 a.m. saying "the site is down."