Skip to content

node_exporter Metrics Missing — Six Symptom Categories, Root-Cause Decision Tree, and Five Real Incidents

Every "my CPU chart used to work three hours ago" panic traces back to one mental trap: metrics don't vanish, they stop being exported. That flips debugging from "what's wrong with Prometheus?" to "who decided not to say the word?" This is the six-symptom tree from my last three clusters, plus five incidents that taught me the lessons.

1. Why metrics "disappear" feels like magic

Every case falls into one of six buckets:

  1. A collector was disabled by flag
  2. The /proc or /sys path it needs is unreadable
  3. K8s hostPath mounts are misconfigured
  4. The kernel doesn't expose the field
  5. A filesystem/device filter regex got too greedy
  6. Prometheus's scrape_timeout fired before the response finished

None is magical. They're structural. Stop chasing ghosts.

2. The 30-second smoke test

Two commands, run on the affected node.

# Total /metrics lines
curl -s localhost:9100/metrics | wc -l

# Metric declarations (HELP/TYPE lines)
curl -s localhost:9100/metrics | grep '^#' | grep -c node_

Reference numbers:

  • Healthy 64-core x86 node_exporter: 5,000–9,000 lines on /metrics
  • ~600–900 HELP/TYPE declarations under node_
  • Below 4,000 lines: at least one collector is broken
  • Declaration count drops 200+ after a deploy: a metric family is gone

The trick that saved me three hours: diff against a known-good node.

curl -s node-01:9100/metrics | grep '^#' > /tmp/good.txt
curl -s node-02:9100/metrics | grep '^#' > /tmp/bad.txt
diff /tmp/good.txt /tmp/bad.txt | grep '^<' | wc -l

Diff told me node_filesystem_* was missing on node-02. Two minutes instead of three hours.

3. Six symptom categories

Six failure domains cover 95% of "metric vanished" tickets.

Symptom Collector flag Reads from Top 3 ways it breaks
node_filesystem_* --collector.filesystem /proc/self/mounts Include/exclude regex typo; fs-types-exclude matches overlay; container can't see host mounts
node_cpu_seconds_total --collector.cpu /proc/stat Kernel < 2.6 lacks steal/iowait; VM with fewer vCPUs than host; /proc remounted with different options
node_memory_* --collector.meminfo /proc/meminfo Kernel < 3.14 has no MemAvailable; container cgroup hides host meminfo; collector explicitly disabled
node_disk_read_bytes_total --collector.diskstats /proc/diskstats, /sys/block/* devices-include excludes dm-/loop; kernel field-layout change; /sys/block not mounted
node_network_receive_bytes_total --collector.netdev /proc/net/dev device-exclude filters veth/cali; lo removed; container namespace hides host NICs
node_netstat_Tcp_* --collector.netstat /proc/net/netstat Kernel < 4.0 lacks newer TCP fields; /proc/net/netstat unreadable; overlayfs hides /proc

diskio and netstat deserve special attention — those fields are kernel-dependent, not exporter-dependent. A metric that exists on the fleet average can be entirely absent on half the nodes. Not a bug, a distribution problem.

4. Root-cause decision tree

[Metric missing]
  │
  ├─► [1] Prometheus target UP?
  │     ├─ NO ──► scrape/network/timeout (Incident 4)
  │     └─ YES ↓
  ├─► [2] `wc -l /metrics` = ?
  │     ├─ < 3000 ──► exporter broken, go to [3]
  │     └─ > 5000 ↓
  ├─► [3] Collector disabled?
  │     `ps -ef | grep node_exporter | grep -o 'no-collector\.[a-z_]*'`
  │     ├─ yes ──► re-enable it
  │     └─ no ↓
  ├─► [4] /proc readable by exporter user?
  │     `sudo -u node-exporter cat /proc/stat`
  │     ├─ permission denied ──► fix perms (Incident 5)
  │     └─ ok ↓
  ├─► [5] K8s hostPath mounts present?
  │     `kubectl get ds -o yaml | grep hostProc`
  │     ├─ missing ──► fix DaemonSet (Incident 2)
  │     └─ ok ↓
  ├─► [6] Kernel old? `uname -r`
  │     ├─ < 4.0 ──► alert with `absent()` fallback
  │     └─ ≥ 4.0 ↓
  ├─► [7] Filter regex greedy?
  │     `ps -ef | grep -E 'mount-points|device-(in|ex)clude'`
  │     └─► test with `echo "path" | grep -P 'REGEX'` (Incident 3)
  └─► [8] scrape_timeout tight?
        └─► see Incident 4

5. Incident 1 — Ubuntu 22.04.4 kernel update killed diskio

Symptom. 6am Tuesday, PagerDuty fired NodeDiskIOAbsent on 40% of the fleet. node_disk_read_bytes_total flatlined.

Timeline. Monday's patch bumped kernel 5.15.0-76 → 5.15.0-83. The new kernel restructured /proc/diskstats, and node_exporter 1.6.0 choked:

node_exporter[24518]: failed to parse /proc/diskstats:
  unrecognized field layout at line 4
node_exporter[24518]: collector diskstats disabled: parse error

Fix. Rolling update to node_exporter 1.8.1 via GitOps took 47 minutes. What I'd do differently: pin the kernel and the exporter version together in one change window. Patching one without the other is how this happens.

6. Incident 2 — K8s ServiceMonitor missed the hostProc mount

Symptom. 200-node EKS cluster, node_memory_* metrics only showed 4% of nodes. Dashboards looked "mostly fine" — classic silent failure.

Root cause. A Helm diff between two renders dropped a single volume:

# WRONG (silently broken across 200 nodes)
volumeMounts:
  - name: hostSys
    mountPath: /sys
    readOnly: true
# MISSING: hostProc

Without /proc mounted, node_exporter read its own container's /proc — empty. Every /proc/*-based collector returned zeros, no error log.

The fix.

volumes:
  - name: hostProc
    hostPath: {path: /proc}
  - name: hostSys
    hostPath: {path: /sys}
  - name: hostRoot
    hostPath: {path: /}
containers:
  - name: node-exporter
    image: quay.io/prometheus/node-exporter:v1.8.1
    args:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/host/root"
      - "--web.listen-address=:9100"
    volumeMounts:
      - {name: hostProc, mountPath: /host/proc, readOnly: true}
      - {name: hostSys,  mountPath: /host/sys,  readOnly: true}
      - {name: hostRoot, mountPath: /host/root, readOnly: true}
    securityContext: {privileged: true}

The three --path.* flags are non-negotiable. Without them, mounting /proc to a different location doesn't help — node_exporter reads /proc directly.

7. Incident 3 — filesystem collector filtered everything after a regex typo

Symptom. A junior engineer filed a ticket: "All my mounts disappeared from Grafana after a config push."

The diff:

# Old:
--collector.filesystem.mount-points-include='^(?!/dev|/proc|/sys|/run)'
# New:
--collector.filesystem.mount-points-include='^/?(dev|proc|sys|run)$'

Two changes compounded: dropping the negative-lookahead turned it into an allow-list of system paths only, and the $ anchor matched the exact strings dev, proc, sys, run — none of which were real mount paths. Zero mounts passed the filter.

How we caught it. The /probe page didn't complain — node_exporter treats an empty mount list as valid. Only a diff revealed it:

curl -s :9100/metrics | grep node_filesystem_size_bytes | wc -l
# before: 47
# after:  0

Rule I teach new hires: test regex changes with echo first.

echo "/var/lib/mysql" | grep -P '^(?!/dev|/proc|/sys|/run)'    # match — good
echo "/var/lib/mysql" | grep -P '^/?(dev|proc|sys|run)$'        # no match — bug

8. Incident 4 — scrape_timeout hit on a full-disk filesystem node

Symptom. One node stopped reporting node_filesystem_* for 40 minutes, then recovered. No exporter logs, no K8s events. Prometheus marked the scrape success — but the data was partial.

Root cause. The node had 1,000+ mounts from a runaway NFS client. Each mount added a syscall to the filesystem collector's parse; each syscall on a full disk took 200–500 ms. The scrape took 42 seconds. Prometheus's default scrape_timeout: 30s fired, cut the connection, kept whatever bytes had been flushed.

level=warn caller=scrape.go:1234 component="scrape manager"
scrape_pool=node scrape_timeout="30s" elapsed="42.3s"
err="context deadline exceeded"

Fix. Two changes in parallel:

scrape_configs:
  - job_name: 'node'
    scrape_interval: 60s
    scrape_timeout: 45s   # was 30s default
    metrics_path: /metrics

And the underlying cleanup: --collector.filesystem.mount-points-exclude='^(?!/|/data|/home|/var|/etc|/opt|/usr)' to keep mounts under 50. The full-disk cleanup was a separate postmortem — 3 years of stale NFS auto-mounts.

9. Incident 5 — systemd collector missing because node_exporter ran as nobody

Symptom. Three weeks into our "run-as-non-root" hardening project, someone noticed systemd_unit_state wasn't firing for failed units. Alerts for stuck services had been silent for 21 days.

Root cause. Our hardened unit dropped User=root; systemd applied the default User=nobody. The systemd collector talks to unix:/run/dbus/system_bus_socket — world-readable, but D-Bus rejects unprivileged callers to system units.

The one-line log nobody had thought to watch:

level=error ts=2025-08-14T09:00:12Z collector=systemd
err="org.freedesktop.systemd1.Manager.ListUnitFiles: access denied"

Fix. Back to root with scoped capabilities:

[Service]
User=root
AmbientCapabilities=
CapabilityBoundingSet=CAP_SYS_TIME CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH
ProtectHome=read-only
ProtectSystem=strict
ReadWritePaths=/var/lib/node_exporter/textfile_collector

The alert that would have caught this earlier:

absent(node_systemd_unit_state) for 15m

Lesson: before shipping a change to the exporter's user, add an absent() alert on the metrics that user-dependent collectors emit.

10. Checklist: 12 commands to audit your node_exporter fleet

# 1. Enabled collectors?
ps -ef | grep node_exporter | grep -oE 'no-collector\.[a-z_]+|collector\.[a-z_]+' | sort -u

# 2. HELP/TYPE declaration count
curl -s localhost:9100/metrics | grep '^# HELP' | grep node_ | wc -l

# 3. Total line count (expect 5000-9000)
curl -s localhost:9100/metrics | wc -l

# 4. What does /probe say?
curl -s localhost:9100/probe | grep -A2 -iE 'error|fail'

# 5. Mounts the filesystem collector should see
awk '{print $2}' /proc/self/mounts | sort

# 6. Can exporter user read /proc?
sudo -u $(ps -o user= -p $(pgrep -f 'node_exporter --web' | head -1)) \
  cat /proc/stat | head -1

# 7. Kernel version — some fields only exist from 4.x+
uname -r

# 8. systemd collector sanity
curl -s localhost:9100/metrics | grep node_systemd_unit_state | wc -l

# 9. Is /proc mounted from the host?
ls -la /proc | head -3

# 10. DaemonSet hostPath check (K8s)
kubectl get ds -n monitoring -o yaml | grep -E 'hostProc|hostSys|hostRoot'

# 11. ServiceMonitor ports (K8s)
kubectl get servicemonitor -n monitoring -o yaml | grep -A5 'ports:'

# 12. Prometheus scrape health
curl -s localhost:9090/api/v1/targets | jq \
  '.data.activeTargets[] | select(.labels.job=="node") | {instance:.labels.instance, health:.health, lastError:.lastError}'

If all 12 check out and metrics are still missing, look at Prometheus-side relabel_configs next — that's how I've seen metrics silently dropped at the scrape target. Not the exporter's problem, but worth grepping before paging anyone.