A Complete Postmortem of a DNS Resolution Failure

Q: The whole site is unreachable — how do I confirm it is DNS and not the web server?

Ping the IP first. If the IP is reachable but the name does not resolve (ping: cannot resolve), this is a resolution-layer problem. Then compare dig +short against your local resolver versus dig @8.8.8.8 — if the public resolver answers but your local one does not, the fault is in the resolver/upstream path, not your web server.

Q: dig returns an IP but the browser still cannot open the site, why?

Layered caches. The browser, the OS, and the local resolver each keep their own cache, and dig only queries the system resolver — it does not flush the browser. Confirm the authoritative answer with dig +noall +answer, then clear the browser cache and the OS DNS cache, and rule out a hosts entry or a proxy/DoH override.

Q: TTL cuts both ways — how does it help pinpoint a fault?

TTL is the lifetime of a resolved record. A short TTL amplifies authoritative flapping; a long TTL keeps clients on a stale record long after the fix. Look at the TTL value dig returns against the timeline: if the resolver is serving a pre-fault value and its TTL has not expired, caching is masking the upstream change — which is exactly why you must flush instead of waiting.

Q: Which line of dig +trace reveals the failure point?

Watch the SERVER and result at each hop, from the root (.) to the TLD to the authoritative servers. The level that starts returning SERVFAIL or times out is where the fault is. If +trace dies at the last hop but dig @8.8.8.8 works, the problem is usually in your authoritative configuration — NS records, SOA, or a broken DNSSEC chain.

Q: How do I verify resolution is truly back after the fix?

Three layers: 1) dig @8.8.8.8 domain +short returns the expected IP; 2) dig +trace domain succeeds through the final authoritative hop; 3) the local dig domain +short resolves. Then flush caches — sudo resolvectl flush-caches on systemd-resolved, sudo rndc flush on BIND — and dig once more to confirm no negative cache remains.

This reconstructs one representative class of "the whole site suddenly went down." Your domain and servers will differ, but the diagnostic ladder and the traps are universal: separate the resolution layer from the web layer, climb the resolver chain, and do not let TTL caching fool you.

Summary

Starting in the morning, a production site began timing out from some regions; within half an hour it was fully unreachable. The browser showed ERR_NAME_NOT_RESOLVED, and ping on the domain returned cannot resolve. This was not a web-server problem — the name itself stopped resolving.

Impact

Every externally reachable domain was affected, not just one subdomain.
Services reached by raw IP were unaffected — the first split signal: whatever worked by IP was fine; whatever used a name was down.
Roughly 40 minutes from the first user report to recovery after an explicit cache flush.

One useful way to frame this: the outage had nothing to do with application code. No deploy had gone out the night before, no config had changed, and the load was normal. The only thing that changed was that a name stopped resolving — which is exactly why a resolution-layer check is the fastest first move. It keeps you from debugging a healthy web stack while the real problem sits a level lower.

Timeline

Time	Event
T+0	First region reports timeouts
T+15	Grows to fully unreachable; ping reports cannot resolve
T+20	Confirmed resolution-layer fault; dig ladder begins
T+32	Upstream authoritative identified as unhealthy; fix starts
T+38	Upstream restored; direct public-resolver check passes
T+40	Explicit local cache flush; site recovers

Root cause

The real trigger was upstream authoritative DNS: one of the domain's authoritative NS servers returned SERVFAIL for a subset of queries during the fault window. Two amplifiers turned it into a full outage:

TTL cache masking: recursive resolvers had cached the pre-fault A record and kept serving it until TTL expiry; after the upstream recovered, they kept walking the negative cache, so the recovery signal was delayed.
Single-authoritative reliance: the NS set was thin, with no geographic or network diversity, so one flaky authoritative affected everything.

The decisive clue (and the watershed of the whole postmortem): during the fault, dig @8.8.8.8 domain answered directly, but the default resolver path did not — the fault was in the authoritative/recursive chain, not in your web stack.

Diagnostic traps that actually cost time

What slowed recovery was not "couldn't find it" — it was ordering and caching noise. Real lessons:

Ping the IP before ping the name. My first reflex was ping example.com; on cannot resolve I nearly restarted the web service. Calming down and running ping <web_ip> first — IP reachable, name not — is what locked it to the resolution layer. Reverse the order and you waste half a day.
Never trust a single resolver. dig example.com goes through your system resolver and may return a cached value. Cross-check with dig @8.8.8.8 and dig @1.1.1.1 to separate "local resolver problem" from "upstream authoritative problem."
Negative caching lies too. During the outage, recursive resolvers cache SERVFAIL or NXDOMAIN for a window set by the SOA negative TTL. After the upstream heals, those bad results keep being served until expiry — another reason fixing the upstream alone is not enough and a manual flush is mandatory.
Hosts entries and proxy/DoH hijack. If dig @8.8.8.8 also fails but some pages open, suspect /etc/hosts, browser DoH (encrypted DNS), or a system proxy rewriting resolution. getent hosts reveals whether the hosts file is overriding.
DNSSEC failure masquerades as latency. A domain with a broken DNSSEC chain often shows as "intermittently times out, sometimes works," easily mistaken for flapping. dig +dnssec +noall +answer and checking the ad flag rules it out fast.

Fix

Climb the ladder, elimination first:

# 1) Separate the layers: ping the IP, then try to resolve the name
ping -c2 <web_ip>            # reachable -> web layer probably fine
ping -c2 example.com          # cannot resolve -> resolution layer
getent hosts example.com      # empty -> the OS cannot resolve it

# 2) Bypass your local resolver, hit a public resolver directly
dig @8.8.8.8 example.com +short     # has IP -> local/upstream recursive issue
dig @1.1.1.1 example.com +short

# 3) Climb to the authoritative servers; whichever level dies is the fault
dig +trace example.com | tail -20
# watch the last hop: SERVFAIL / connection timed out -> authoritative outage

# 4) Inspect SOA / NS / DNSSEC state
dig example.com SOA +noall +answer
dig example.com NS +noall +answer
dig example.com +dnssec +noall +answer

For immediate containment, point the local resolver at a healthy upstream (in /etc/resolv.conf or systemd-resolved), or temporarily pin the domain in /etc/hosts to the known IP so the business recovers first.

For the permanent fix, repair the authoritative side (restore NS availability, correct the NS/SOA configuration, confirm the DNSSEC signature chain is complete). After the fix, flush caches explicitly — do not wait out the TTL:

# systemd-resolved
sudo resolvectl flush-caches
# BIND / named
sudo rndc flush
# verify
dig example.com +short

Prevention

Three things this incident exposed, each with a guard:

Multi-authoritative, multi-line redundancy: at least two authoritative servers across different networks/DCs, to remove the single point.
DNSSEC and SOA health checks: a scheduled dig +dnssec +trace to verify the signature chain; keep SOA refresh/retry from being too aggressive so authoritative flapping is not amplified.
Resolution-layer monitoring: monitor the resolution itself, not just the web site — a scheduled dig domain +short compared against the expected IP, alerting on TTL change, SERVFAIL, or latency spikes. This is the true first layer of "the site is down."
TTL policy: keep TTL on critical records short (300-600s) to shrink the stale-cache window after a fix; pair with a flush script that clears caches when recovery is declared.
Failure drill: periodically simulate "authoritative DNS unreachable" and verify the monitor alerts within T+5 and the failover path actually works.

A minimal resolution-layer monitor — one step ahead of "the site is down":

# Resolve once a minute and compare against the expected IP
cat > /usr/local/bin/check-dns.sh <<'EOF'
#!/bin/bash
DOMAIN=example.com
EXPECT=203.0.113.10
GOT=$(dig +short @8.8.8.8 "$DOMAIN" | head -1)
if [ "$GOT" != "$EXPECT" ]; then
  echo "DNS MISMATCH: $DOMAIN expected $EXPECT got ${GOT:-EMPTY}" | mail -s "DNS alert" [email protected]
fi
EOF
chmod +x /usr/local/bin/check-dns.sh
# crontab: */1 * * * * /usr/local/bin/check-dns.sh

The value is not complexity — it is making resolution itself a monitored object. It catches "the name resolves to the wrong thing," a layer that usually exposes an upstream fault before a web probe does. A richer version adds dig +time=2 +tries=1 for timeout control and records TTL drift as a trend.

Lesson

For "the whole site is down," separate the resolution layer from the web layer first — if it works by IP but the name will not resolve, it is DNS. The diagnostic order is always: local resolver → public resolver → dig +trace authoritative; whichever level dies first is where the fault lives. The thing that is easiest to miss is TTL caching: it masks the truth during the fault and delays recovery after the fix. So flush caches explicitly after the fix, and monitor resolution itself rather than waiting for users to report the site down.

The most valuable takeaway is simple: if you monitor the wrong object, you always react half a beat late. We used to "monitor the site," which only alerted when the web layer fully failed. Now we "monitor the resolution result," which surfaces the fault the moment the authoritative starts flapping. Moving the monitor one layer up shortens MTTR more reliably than writing another alert rule — that, more than any single command, is what this incident taught me.

It also reinforced that the cheap diagnostic is usually the right one: three dig invocations, run in the right order, told us more about where the fault lived than any dashboard. When a whole site dies, resist the urge to restart services you cannot blame. Run the ladder first, flush second, and only then declare the incident resolved.