Skip to content

A Complete Postmortem of a Certificate Expiring Unmonitored

This reconstructs one representative class of "the site certificate expired, browsers showed a red screen, and the renewal failure went unnoticed." Your CA and domain will differ, but the diagnostic ladder and the traps are universal: confirm the real expiry first, reproduce why renewal failed, then make both "expiry" and "renewal failure" monitored objects.

Summary

On a Monday morning rush, users reported the site was red-screened in the browser: NET::ERR_CERT_DATE_INVALID, the certificate had expired. Oddly, it had been fine the day before — Let's Encrypt renews every 90 days automatically, so it should never expire. openssl s_client showed notAfter was already 3 days in the past, meaning the renewal failure had happened at least 3 days earlier with no monitoring noticing.

Impact

  • Every HTTPS request was blocked by the browser; users saw the "connection not secure" red screen directly.
  • HTTP still worked (if port 80 remained open), but nearly all traffic was HTTPS, so the site was effectively down.
  • Roughly two hours from discovery to a freshly issued certificate and a CDN cache refresh.
  • Because the site itself was reachable (the homepage returned), the external reachability monitor stayed green — no alert at all.

Timeline

Time Event
D-90 Last successful renewal; expiry set to day D
D-45 Normal renewal window (Let's Encrypt renews 30 days out) should have fired
D-3 Certificate actually expired (renewal silently failed, no alert)
D+0 08:30 Users report the browser red screen; investigation begins
D+0 09:00 openssl s_client confirms notAfter is in the past
D+0 09:20 certbot renew --dry-run reproduces the failure; HTTP-01 challenge blocked
D+0 10:00 Challenge path fixed; certbot renew --force-renewal reissues
D+0 10:30 CDN cache refreshed; HTTPS restored; expiry alerts added

Root cause

The real trigger was the renewal job silently failing one step before validation succeeded: the ACME HTTP-01 challenge path /.well-known/acme-challenge/ was blocked by a CDN/WAF rule, so Let's Encrypt's validation request could not fetch the token and the renewal failed. Two amplifiers turned it into a full outage:

  1. The renewal failure was silently swallowed: the renewal job (certbot's systemd timer) only logged each failure — no exit-code monitoring, no failure alert. It failed for 45 days without anyone knowing.
  2. No certificate-expiry monitoring: we monitored "site reachability" but not "certificate validity." The site returned 200 the whole time, so there was no signal before expiry.

The decisive clue (and the watershed of the whole postmortem): openssl s_client showed notAfter in the past, and certbot renew --dry-run reproduced the exact same challenge failure — the problem was in the validation step of the renewal chain, not in the certificate itself.

Diagnostic traps that actually cost time

What slowed recovery was not "couldn't find it" — it was treating "site reachable" as "site healthy". Real lessons:

  1. Confirm the real expiry first. A browser red screen can also come from an incomplete chain, a hostname mismatch, or a missing intermediate — not necessarily "expired." Parse notAfter with openssl s_client -servername and compare to now; do not take the browser warning as the conclusion.
  2. Without -servername you read the wrong cert. On a CDN or shared IP, omitting SNI pulls the default site's certificate and gives a wrong "not expired." Always pass -servername when diagnosing a domain cert.
  3. certbot renew is a check, not a renewal. renew only acts within 30 days of expiry; run it early and it does nothing. Reproduce the full validation flow with renew --dry-run to find the failing step.
  4. Reachability 200 is not certificate validity. A site being reachable does not mean HTTPS is healthy. Monitor "service reachable" and "certificate valid" as two separate layers, never one probe.
  5. Watch the renewal exit code. certbot failure may only be a log line with a non-zero exit code and no system error. Make the renewal job's exit code an alert signal — you should know on the day it fails, not the day the cert expires.

Fix

Climb the ladder, confirm the expiry first:

# 1) Confirm whether it really expired (must pass -servername, or a CDN serves the wrong cert)
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
# compare notAfter with now; in the past -> confirmed expired

# 2) List renewal state for all certificates
certbot certificates
# each cert's Expiry Date / Renew status at a glance

# 3) Reproduce the renewal failure to find the failing step
certbot renew --dry-run
# read the log: did HTTP-01 or DNS-01 fail?

# 4) If HTTP-01, probe the challenge path
curl -I http://example.com/.well-known/acme-challenge/test
# blocked by CDN/WAF -> allowlist the path; 404 with no CDN block -> check the web server config

For immediate containment, renew manually once the challenge path is fixed:

# Allow the HTTP-01 path (CDN/WAF rule + web server location)
# nginx example:
# location ^~ /.well-known/acme-challenge/ { root /var/www/letsencrypt; }
# Then renew manually
certbot renew --force-renewal
# Verify the new cert is issued
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

For the permanent fix, add two layers of monitoring + watch the renewal exit code:

# Alert 30/14/7 days before expiry (black-box TLS idea)
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -enddate \
  | cut -d= -f2 | while read -r d; do
      let days=($(date -d "$d" +%s)-$(date +%s))/86400
      echo "CERT EXPIRES IN $days DAYS (example.com)"
      [ "$days" -le 30 ] && echo "ALERT: cert expiring" | mail -s "Cert alert" [email protected]
    done

Prevention

Three things this incident exposed, each with a guard:

  1. Certificate-expiry monitoring (first guard): do not monitor site reachability only — monitor certificate validity. Parse notAfter periodically with openssl s_client or a black-box TLS probe (such as blackbox_exporter's tls probe) and alert at 30/14/7 days.
  2. Renewal-failure monitoring: schedule certbot renew --dry-run and check the exit code, alerting on non-zero. This surfaces a broken renewal chain weeks before the expiry date.
  3. Challenge-path allowlist and regression: after allowlisting the HTTP-01 path, fold it into automated regression (periodically curl the path and verify 200) so a later CDN/WAF change cannot silently block it again.
  4. Certificate dashboard: summarize certbot certificates into a dashboard listing each cert's expiry date and days remaining, so the one about to expire is obvious at a glance.
  5. Failure drill: periodically simulate a renewal failure and verify both the expiry alert and the renewal-failure alert fire within the expected window.

Lesson

For a "browser red screen," confirm the real expiry first — parse notAfter with openssl s_client -servername, do not take the browser warning as the conclusion. The diagnostic order is always: confirm expiry → certbot certificates → renew --dry-run to reproduce the failure → fix the challenge path. The thing that is easiest to miss is "renewal failure silently swallowed": certbot logs a failure without surfacing it, so without exit-code monitoring nobody knows.

The most valuable takeaway is simple: if you monitor the wrong object, you always react half a beat late. We used to "monitor site reachability," which only signaled once the browser red-screened. Now we "monitor certificate validity and the renewal exit code," which surfaces the problem weeks before the expiry date. Moving the monitor one layer up shortens MTTR more reliably than writing another alert rule — that, more than any single command, is what this incident taught me.

It also reinforced that the cheap diagnostic is usually the right one: one openssl s_client invocation told us more about where the fault lived than guessing from the browser warning. When a certificate appears expired, confirm it, reproduce the renewal failure, fix the validation path, and only then declare the incident resolved.

Related reading: [SSL/TLS Handshake Troubleshooting../../network/ssl-tls-handshake-troubleshoot.md) · [DNS Troubleshooting../../network/dns-troubleshoot.md) · [CDN Principles../../network/cdn-principles.md)