A Complete Postmortem of a Log Partition Filling Up

Q: The service process is running but features seem broken — how do I tell if it is disk?

Run df -h first. If /var/log or the data partition is at 100%, many services fail silently when they cannot write logs or temp files — the process does not crash, but new requests error out and writes fail. Probe with a test write to the affected partition to confirm it is full or read-only.

Q: df shows 100% but du does not add up to the partition size, why?

Classic 'deleted but not released'. A process still holds an open handle on a deleted log file (common after logrotate without a reload, or in long-running processes), so the filesystem keeps the space while du can no longer see it. Use lsof +L1 or lsof | grep deleted to find the holder, then restart it or make it reopen the file.

Q: How do I quickly find which log is exploding?

Run du -sh /var/log/* 2>/dev/null | sort -h and look at the largest entries, then sample the suspicious file with du -sh twice and observe the growth rate. If a single log keeps growing, an app is spamming error stacks or debug output — investigate why the app keeps logging.

Q: I configured logrotate but the disk still filled, why?

Four common reasons: 1) the rotation interval is too long for the volume; 2) the disk was already full before compress ran; 3) the process never got a reopen signal (copytruncate conflicts with SIGHUP), so the handle stays on the old file; 4) some log path is not covered by any rule. A dry run with logrotate -d shows which rules fire and which files are untouched.

Q: Besides adding monitoring, what else should I do to prevent a recurrence?

A three-part guard: monitoring (85% warn / 90% critical), a complete logrotate rule set, and a log-growth dashboard. Also put logs on their own partition, rate-limit per-process debug output, and alert on log-write failures themselves (via journald or a log agent).

This reconstructs one representative class of "/var/log filled up and the service failed without crashing." Your volume might be a data disk or a different path, but the diagnostic ladder and the traps are universal: a running process is not a healthy service, so look at the disk first, find which log is exploding, and do not let "deleted but not released" fool you.

Summary

Starting in the afternoon, a core application became "intermittently broken": some endpoints returned 500, database writes failed, but the process never stopped and CPU stayed normal. It looked like a transient fault; several restarts did not help. A single df -h was the turning point — the root filesystem was at 100%, and the separate /var/log partition was at 100% too. Every broken feature was, underneath, a failed write to disk.

Impact

Every write operation failed: log writes, temp-file writes, buffered writes. Endpoints returned 500, database writes errored, scheduled jobs reported failures.
Read operations (queries, config loads) mostly kept working, so the system looked "half broken, half fine" — the most misleading part of the whole incident.
Roughly three hours from the first intermittent 500 to the point where the partition was cleaned and writes recovered.
Because the disk was full, the alerting logs could not be written either; the monitoring platform only saw "more 500s," with no direct "disk full" signal.

Timeline

Time	Event
T+0	First intermittent 500; a service restart briefly helps
T+45	More 500s and DB write errors; suspicion turns to an app bug, code review begins
T+90	Process confirmed alive; investigation shifts to the system layer
T+120	`df -h` reveals /var/log at 100%; disk identified as the cause
T+150	`du -sh /var/log/* \\| sort -h` finds the exploding log; an app error loop is the source
T+165	Old logs cleaned and logrotate triggered; writes recover
T+180	Dedicated disk-usage alert added to /var/log; postmortem closes

Root cause

The real trigger was a log-volume explosion: during the fault window the app entered an "error retry loop," logging a full error stack on every retry, so /var/log/app/error.log grew at tens of MB per minute and filled the /var/log partition. Two amplifiers turned it into a full outage:

Logrotate did not cover that path: the rotation config matched only the generic /var/log/*.log; the app's own app/error.log matched no rule, so it was never rotated — writes went in but nothing came out.
No disk-usage monitoring: we monitored endpoint availability but not "free disk space." By the time the 500s appeared, the disk had been full for a while.

The decisive clue (and the watershed of the whole postmortem): df -h showed /var/log at 100%, and touch /var/log/test returned No space left on device — this was a write failure, not an application logic bug.

Diagnostic traps that actually cost time

What slowed recovery was not "couldn't find it" — it was pointing at the wrong layer. Real lessons:

Look at the disk before the code. My first instinct was an app bug; I read code for over an hour. A single df -h later settled it. For "intermittently broken" services, check the system layer first, not the business logic.
A running process is not a healthy service. Many services degrade silently when log writes fail — they do not exit, they just error on new requests. Judging "service OK" from ps alone misses the whole disk class of failure.
du does not add up. df says 100%, but du -sh /var/log/* totals less than the partition — that is "deleted but not released." A process still holds the handle on the deleted log; only lsof +L1 reveals it.
Restart is not a fix. Restarting the service just reopens the log file and temporarily releases the handle; if the log is still exploding, the disk fills again. Treat the log source, not the process.
Rotation must cover the real path. Matching only /var/log/*.log misses app subdirectories. Run logrotate -d as a dry run and check which files are never handled.

Fix

Climb the ladder, elimination first:

# 1) Look at the disk — first move for "intermittently broken"
df -h
# If / and /var/log are both at 100%, this is almost certainly a write failure
touch /var/log/test 2>&1        # No space left on device -> confirm full

# 2) Find which log is exploding
du -sh /var/log/* 2>/dev/null | sort -h | tail
# Sample the suspicious file to observe the growth rate
du -sh /var/log/app/error.log
sleep 30 && du -sh /var/log/app/error.log   # grew -> still logging

# 3) Find handles on deleted files
lsof +L1 2>/dev/null | grep -iE 'log|deleted' | head
# A deleted entry -> the process still holds the file; restart or reopen

For immediate containment, delete old logs to free space (keep the most recent copy), then trigger rotation manually:

# Keep the last 3 copies, delete the rest (confirm nothing is using them first)
find /var/log -name '*.log' -mtime +3 -delete
# Manual rotation (no compress yet — free space first)
logrotate -f /etc/logrotate.conf
# Release handles on deleted files: restart the holder
systemctl restart <the-app>
# Confirm writes are back
df -h && echo ok > /var/log/test && rm /var/log/test

For the permanent fix: fix the log source + complete the rotation + add monitoring. First repair the app-side "error retry loop" that was spamming the log, then add a rotation rule for app/error.log:

# /etc/logrotate.d/app
/var/log/app/*.log {
  daily
  rotate 7
  compress
  delaycompress
  missingok
  notifempty
  copytruncate
}

Prevention

Three things this incident exposed, each with a guard:

Disk-usage monitoring (first guard): do not monitor endpoint availability only — monitor free disk space. Add alerts per critical partition: 85% warning, 90% critical, 95% emergency. This is the earliest detector of "the service broke without reporting it."
Complete the logrotate rule set: walk every log path and confirm each has a rule; periodically dry-run with logrotate -d to catch paths that are never handled.
Log-growth dashboard and rate limit: monitor each log's growth rate (MB/min) and alert on abnormal growth; rate-limit app-side debug/error-stack output so a retry loop cannot fill the disk.
Logs on their own partition: put /var/log or app logs on a separate volume so a full log disk cannot take down the root and data partitions.
Failure drill: periodically simulate "disk at 95%" and verify the monitor alerts within T+5 and the cleanup path actually works.

A minimal disk monitor — one step ahead of "endpoints are 500ing":

# Check critical partitions every 5 minutes and alert above the threshold
cat > /usr/local/bin/check-disk.sh <<'EOF'
#!/bin/bash
THRESHOLD=85
df -h | awk -v t=$THRESHOLD 'NR>1 && $5+0 >= t {print $6" "$5" used"}' | \
while read -r line; do
  echo "DISK HIGH: $line" | mail -s "Disk alert" [email protected]
done
EOF
chmod +x /usr/local/bin/check-disk.sh
# crontab: */5 * * * * /usr/local/bin/check-disk.sh

The value is not complexity — it is making free disk space itself a monitored object. It catches "the disk is nearly full," a layer that usually surfaces hours before an application starts erroring. A richer version feeds Prometheus node_exporter's node_filesystem_avail_bytes straight into an alert rule.

Lesson

For "the service broke without reporting it," look at the disk first — the process running and CPU being normal does not mean the service is healthy. The diagnostic order is always: df -h → du -sh to find the exploding log → lsof +L1 for deleted-but-not-released handles → fix the log source rather than restart. The thing that is easiest to miss is "deleted but not released": df is full but du does not add up, which means a process still holds the handle on a deleted file.

The most valuable takeaway is simple: if you monitor the wrong object, you always react half a beat late. We used to "monitor endpoint availability," which only alerted once the business was already erroring. Now we "monitor free disk space," which surfaces the problem hours before the disk fills. Moving the monitor one layer up shortens MTTR more reliably than writing another alert rule — that, more than any single command, is what this incident taught me.

It also reinforced that the cheap diagnostic is usually the right one: a single df -h told us more about where the fault lived than an hour of code review. When a service is "half broken," resist the urge to blame the application. Check the disk first, find the log source second, and only then declare the incident resolved.

Related reading: [Linux Disk IO Monitoring../../monitor/linux-disk-io-monitor.md) · [Loki Log Aggregation../../monitor/loki-log-aggregation.md) · [Prometheus TSDB Operations../../monitor/prometheus-tsdb-operations.md)