Skip to content

title: "node_exporter Metrics Missing: 6 Symptom Categories and 5 Real K8s Incidents" description: "Systematic troubleshooting for when node_exporter metrics go dark or wrong. 6 collector symptom categories, 5 real Kubernetes incidents with root causes, and PromQL recipes to detect missing metrics before dashboards break." tags: [node_exporter, prometheus, monitoring, troubleshooting, kubernetes] lang: en date: 2026-10-26 original: true answer: | Systematic node_exporter troubleshooting: 6 collector symptom categories, 5 real K8s incidents, PromQL recipes to detect missing metrics before dashboards go dark. faq: - q: "Why are my node_exporter filesystem metrics suddenly missing?" a: "Two usual suspects: the exporter is scraping /proc/mounts but the mount is NFS and /proc/mounts returns truncated paths, or the node has 500+ mounts and node_exporter's default -collector.filesystem.mount-points-exclude filter drops them. Check curl localhost:9100/metrics | grep node_filesystem_device_size_bytes and count how many devices show up." - q: "cpu.stat metrics exist but they look stale" a: "This means node_exporter is running but the /proc/stat read is stale — usually because node_exporter is in a container without /proc/host mounted. In K8s, mount /host/proc as read-only into the container. If it's bare-metal, restart the exporter." - q: "How do I detect metrics disappearing BEFORE the alert fires?" a: "Write a recording rule: absent_over_time(node_cpu_seconds_total[10m]) == 1 — fires 10 minutes after the last sample. Pair with increase(up{job="node-exporter"}[1m]) == 0 to catch flapping before it becomes a full outage." - q: "K8s: my ServiceMonitor scrapes 100 nodes but only 30 have metrics" a: "Selector mismatch. Check kubectl get servicemonitor -n monitoring -o yaml | grep matchLabels. Common bug: the ServiceMonitor's selector.matchLabels.app: node-exporter but the Services have app.kubernetes.io/name: node-exporter (different key). Also check RBAC — kube-prometheus-stack ServiceAccounts sometimes lack pod read access to label-discovery." - q: "Difference between cAdvisor and node_exporter?" a: "cAdvisor = container metrics (per-container CPU/mem/fs). node_exporter = node-level metrics (aggregate CPU, disk, network, filesystem). They're complementary. A common mistake is scraping both and getting overlapping cpu/memory labels — set label allowlists in relabel_configs."

node_exporter Metrics Missing: 6 Symptom Categories and 5 Real K8s Incidents

I've debugged "where did this metric go?" on prod twice this year alone, and both times it wasn't the metric that was missing — it was the assumption that we were still looking at the right machine. A node_exporter that is up and returning 200s is not the same thing as a node_exporter that is useful. The target shows green, the dashboard renders, and someone on-call spends twenty minutes wondering why the memory panel says "no data" for a node they know is alive.

That gap between "up" and "useful" is the whole discipline here. A metric that quietly returns a stale or wrong value is worse than a missing one, because a missing metric produces an alert and a stale metric produces a confident lie. Everything below is organized around finding that lie before it does damage.

The 6 Symptom Categories

I group nearly every "node_exporter metric missing" report I get into one of six buckets, aligned with the collector that owns the metric family.

1. node_filesystem_* — mounts vanished

The usual cause is not that the mount is gone — it's that node_exporter's default mount-points-exclude regex is doing its job too well. On a heavily-mounted host, or a node with many bind mounts from a Kubernetes workload, /proc/self/mounts returns a long list, and node_exporter silently filters out most of it. Check the actual output first:

curl -s localhost:9100/metrics | grep '^node_filesystem_device_size_bytes' | wc -l
cat /proc/self/mounts | grep -v -E '^(proc|sysfs|tmpfs|devpts|cgroup|overlay)' | wc -l

If the two numbers disagree, node_exporter is filtering. Loosen with -collector.filesystem.mount-points-exclude='^(proc|sysfs|efivarfs|devpts|devtmpfs|none|auto|usr\.merge|snap|run|dev|sys|run|var/lib/.*)$' — and always pair with -collector.filesystem.fs-types-exclude so you don't accidentally include overlay inside a container.

2. node_cpu_seconds_total — modes missing or values frozen

Two different bugs share the same symptom. First, on older kernels (< 4.x) or non-x86 architectures, steal, iowait, or guest_nice modes may not appear at all — that is a kernel feature, not a node_exporter bug. Second, and more annoying: on a Kubernetes node, if the DaemonSet doesn't mount hostProc, the container reads its own /proc/stat and the values look correct but they're per-cgroup, not per-host. If CPU modes look frozen or the numbers are suspiciously low, run:

kubectl exec -n monitoring ds/node-exporter -- cat /proc/stat | head -3

Compare against kubectl exec into any other pod. If the two don't match, /host/proc is not mounted.

3. node_memory_* — MemAvailable disappears

node_memory_MemAvailable_bytes was added in kernel 3.14. Anything older returns nothing. If you're still on CentOS 7 with a 3.10 kernel, plan around it:

node_memory_MemAvailable_bytes > 0
OR (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) > 0

On K8s, the more common failure is that the container is inside a cgroup and reads /proc/meminfo from the cgroup scope. MemAvailable inside a cgroup-limited container is misleading — it reports what the cgroup thinks is available, not the host. The fix is the same as above: mount /host/proc and pass --path.procfs=/host/proc.

4. node_diskio_* — device renamed

node_diskio derives device names from /proc/diskstats and /sys/block/. If udev rules rename sda to nvme0n1 after a firmware update, or LVM thin pools get renumbered on reboot, your historical time series silently breaks because the label set changed. This is a cardinality problem masquerading as a missing-metric problem. Detect it with:

changes({__name__=~"node_disk_.*_total", instance=~"10.20.."})[1h]

A sudden spike means a device label flipped. The fix is often to normalize labels with a recording rule.

5. node_network_* — interface renamed

Same root cause, different collector. On Debian/Ubuntu, /etc/udev/rules.d/70-persistent-net.rules and the newer systemd-networkd naming (ens3, enp0s3) can silently rename interfaces across reboots. K8s makes it worse: veth*, cali*, flannel* interfaces are created and destroyed constantly, and node_exporter's default device-exclude filters out half of them. I typically end up with:

--collector.netdev.device-include='^(eth|ens|enp|cali|flannel|veth|cni|lo)$'

But then I have to keep the dashboards honest about which interfaces are real vs. ephemeral.

6. node_netstat_* — kernel version gaps

/proc/net/netstat exposes raw TCP/UDP counters, but which counters exist depends on the kernel. Tcp_RetransFail was added in 4.x, Udp_InErrors has been around longer. If you write a PromQL rule that assumes node_netstat_Tcp_RetransFail exists on all your nodes, you'll get alerts on the new fleet and silent non-fires on the old fleet. Never assume a field exists — check /proc/net/netstat on the oldest host in your fleet first.

Incident 1: ServiceMonitor Selector Mismatch

I've seen this exact incident three times. The ServiceMonitor looks perfect:

spec:
  selector:
    matchLabels:
      app: node-exporter

But the Services have:

labels:
  app.kubernetes.io/name: node-exporter

The keys don't match. Prometheus Operator matches Services via ServiceMonitors, not vice versa — so a broken selector means zero targets, not an error. The tell-tale sign is kubectl get servicemonitor -n monitoring -o yaml showing a matchLabels block and kubectl get svc -A | grep node-exporter showing different label keys. Fix by aligning the keys, and add a check in your CI: kubectl get servicemonitor -A -o json | jq '.items[].spec.selector.matchLabels'.

Incident 2: kubelet Scrape Label Bleed

kube-prometheus-stack scrapes both kubelet and node_exporter. The kubelet's /metrics/cadvisor endpoint has labels like pod, container, namespace, but also has node labels derived from kubelet's own view. If you don't relabel, dashboards that show "CPU per node" get double-counted: once from container_cpu_usage_seconds_total and once from node_cpu_seconds_total. The fix is a relabel_configs block in the kubelet scrape config that drops container-level CPU from node dashboards, or a recording rule that computes "node CPU" from only node_exporter.

Incident 3: cAdvisor + node_exporter Double-Counting CPU

Related to #2, but the reverse direction. Some teams scrape kubelet only for "node CPU" and assume cAdvisor is authoritative. On K8s 1.29+, cAdvisor reports per-container CPU, and summing across containers does not equal node CPU because of kernel idle accounting. I've caught this in dashboards that show "105% CPU usage" because both sources were summed. Set a hard rule: node-level CPU always comes from node_cpu_seconds_total, container CPU from container_cpu_usage_seconds_total, and never aggregate across the boundary.

Incident 4: /var/lib/node_exporter Permissions After Reboot

This is the "why did it work last week" trap. On a fresh install, the textfile collector directory is owned by the node-exporter user. After a reboot, if some other process (Ansible, a cron script, a systemd-timer) creates files in /var/lib/node_exporter/textfile_collector/, those files can end up owned by root. node_exporter then logs permission denied on every scrape, but the /metrics endpoint still returns 200 — because textfile_collector is one of many collectors and its failure is not fatal.

Check:

ls -la /var/lib/node_exporter/textfile_collector/
journalctl -u node_exporter --since "10 min ago" | grep -i permission

The fix is to enforce ownership in your config management: chown node-exporter:node-exporter /var/lib/node_exporter/ and add an Ansible file task that keeps it owned by that user.

Incident 5: /proc Mount Removed in a Helm Upgrade

The silent killer in kube-prometheus-stack. Chart version 60.x upgraded the node-exporter DaemonSet and, in one patch release, removed the hostPath mount for /host/proc on the assumption that hostPID: true was sufficient. It's not. hostPID shares the PID namespace; it doesn't share the mount namespace. Every node_exporter pod after that upgrade was reading container-local /proc and silently returning wrong numbers. Dashboards looked green. Alerts fired correctly. But every CPU, memory, and filesystem number was per-cgroup, not per-node.

The fix, in the DaemonSet spec:

volumeMounts:
  - name: proc
    mountPath: /host/proc
    readOnly: true
volumes:
  - name: proc
    hostPath:
      path: /proc
args:
  - --path.procfs=/host/proc

Always diff a Helm upgrade against the previous chart version. helm diff upgrade should be a required gate in your release pipeline.

PromQL Recipes to Catch Missing Metrics Early

These are the rules I keep in every Prometheus install. They fire before a dashboard goes dark.

# Any node_exporter metric family that hasn't reported in 10 minutes
absent_over_time(node_cpu_seconds_total[10m]) == 1

# Up but no data — the target is green but the collector is broken
up{job="node-exporter"} == 1 AND absent_over_time(node_filesystem_size_bytes[10m]) == 1

# Flapping before it breaks
count_over_time(up{job="node-exporter"}[1m]) != 2

# Cardinality explosion — a device/interface was renamed
changes({__name__=~"node_disk_.*_total"}[1h]) > 0

Debug Checklist

Symptom First command Likely cause Fix
/metrics returns 200, dashboard has no data kubectl get sm -n monitoring -o yaml Selector mismatch Align label keys
CPU numbers look wrong but non-zero kubectl exec ds/node-exporter -- cat /proc/stat Missing /host/proc mount Add hostPath mount
MemAvailable missing cat /proc/meminfo on host Old kernel (< 3.14) Use fallback sum rule
Disk metrics missing after reboot ls /sys/block/ udev renamed device Normalize labels
textfile_collector silently fails journalctl -u node_exporter Wrong ownership Enforce via config mgmt
Node CPU > 100% in dashboards Check recording rules Double-counting cAdvisor Scope by source

FAQ

Q: My ServiceMonitor scrapes 100 nodes but only 30 have metrics — why?

Selector mismatch is 90% of these cases. The ServiceMonitor's selector.matchLabels must exactly match a label on the Services (not the Pods) that it wants to monitor. kubectl get servicemonitor -n monitoring -o yaml and kubectl get svc -A -l app.kubernetes.io/name=node-exporter will show you the label keys side by side. The other 10% is RBAC: kube-prometheus-stack's Prometheus ServiceAccount needs list and watch on Services and Endpoints in the target namespaces. If you've restricted the ServiceAccount to only its own namespace, targets in other namespaces will silently not be discovered.

Q: How do I detect metrics disappearing before the alert fires?

Prometheus has a built-in for this: absent_over_time(metric[10m]) returns 1 if the metric hasn't been seen in the last 10 minutes. Wire it into a recording rule:

- record: absent_over_time:node_cpu_seconds_total:10m
  expr: absent_over_time(node_cpu_seconds_total[10m])

Then alert on > 0. Pair it with a flapping detector: count_over_time(up{job="node-exporter"}[5m]) < 5 catches targets that are bouncing between up and down before they settle into a full outage. The 10-minute window is a compromise — long enough to tolerate a brief scrape stall, short enough to fire before a dashboard user notices.

Q: cAdvisor and node_exporter overlap — how do I decide which to trust?

They're not competing. cAdvisor reports per-container metrics; node_exporter reports per-node metrics. The right mental model: if the question is "how much CPU did this pod use?", trust cAdvisor. If the question is "how much CPU does this node have available?", trust node_exporter. Summing container_cpu_usage_seconds_total across all containers and calling it node CPU is wrong because kernel idle time is accounted differently at the cgroup level. Set a team rule: node-level queries must use node_-prefixed metrics, container-level queries must use container_-prefixed metrics, and no PromQL rule should reference both.

Q: My node_network_* metrics show only eth0 — where did the veth and cali interfaces go?

They're being filtered by the default device-exclude regex, which drops ^(veth|docker|br|virbr|lxdbr|wwn|cni|cali) — literally every K8s CNI interface. To include them, pass:

--collector.netdev.device-include='^(eth|ens|enp|cali|flannel|veth|cni|lo)$'
--collector.netdev.device-exclude=''

But be careful: including ephemeral interfaces like veth* will explode your cardinality on every pod create/destroy. I usually include only cali* and flannel* (which are stable across pods) and leave veth* out.

Q: node_exporter says up == 1 but /metrics returns almost no data. What's happening?

You're looking at a container-local view. On K8s, node_exporter needs four things to see host data: hostNetwork: true, hostPID: true, hostPath mounts for /proc, /sys, and / (all read-only), and the --path.procfs=/host/proc and --path.sysfs=/host/sys flags. Miss any one and the exporter runs happily, reports 200s, and returns container-scoped metrics that look plausible. The diagnostic is kubectl exec -it ds/node-exporter -- cat /proc/meminfo — if MemTotal matches the container's cgroup limit instead of the host's physical RAM, you're inside the wrong namespace.