Postmortem Library

A postmortem is the rarest and most Google-trusted form of ops content: it does not teach a feature, it reconstructs one incident — when it happened, what broke, how you found it, how you fixed it, and how you stop it recurring. Every case here is written from a first-person operator's viewpoint.

⚠️ Honesty note: these are written as representative incident reconstructions — they reproduce the typical shape and the full diagnostic path of each failure class. They do not name a real company, person, or exact date; the value is the reusable troubleshooting method.

Fixed postmortem template

Every case follows the same seven-part structure:

1. Summary — the symptom and its scale in one line
2. Impact — who was affected, how badly, for how long
3. Timeline — key points from first alert/user report to recovery
4. Root cause — the real trigger, not the surface symptom
5. Fix — how you isolated, patched, and fully resolved it
6. Prevention — the monitoring, alerting, drills, and guards to add
7. Lesson — one takeaway you can reuse

First batch (5 cases)

Case	Value	Column
DNS resolution failure: upstream outage + TTL cache stacking	Whole site suddenly unresolvable; dig traced it level by level to the upstream	Network
Log partition full: /var/log at 100%, services silently failing	Logs filled the disk; the service "failed without an error"	Monitoring
Cert expiry missed: Let's Encrypt renewal failure	Found only on the day it expired; openssl chain check + alerting	Network
HA MQTT discovery failure: topic drift / duplicates	Devices vanished; mosquitto_sub caught the topic drift	Home Assistant
LLM inference OOM: vLLM VRAM leak + overcommit	Inference restarted repeatedly; nvidia-smi found the VRAM blowup	AI

Why a dedicated library

Tutorials teach how to use; postmortems teach what to do when it breaks. Incidents never run out and the form is a durable second content engine; incident reconstructions carry high E-E-A-T and win featured snippets and AI overviews more easily than generic tutorials.

---

[Browse all articles →../../archive.md)