Windows Server Monitoring: wmi_exporter, Prometheus, and Grafana

Q: Why does my disk queue metric show a false alarm on the system drive?

Logical disk metrics carry a volume label like C: or D:. Without a drive letter filter, every mounted volume (including the hidden System Reserved one) is summed. Filter with `windows_logical_disk_idle_seconds{volume="C:"}`.

For operations engineers who manage Windows Server fleets and want real, queryable metrics instead of a screenshot of Task Manager. The takeaway: wmi_exporter gets you from netstat and perfmon to a full Prometheus/Grafana stack in about thirty minutes, but the traps are in the collector defaults and Windows Firewall, not in the installer.

I run a mixed fleet. Linux gets node_exporter; Windows gets wmi_exporter. They are not symmetric. On Linux, the exporter is a single static binary that reads /proc. On Windows, wmi_exporter shells out to WMI, and WMI has permissions, providers, and a cardinality profile that will surprise you on the first heavy box. This article is the path I wish I had on day one: install, collectors, firewall, scrape, dashboard, alerts, and five incidents that all cost me an on-call shift.

1. Pick a Deployment Path: MSI or Zip + NSSM

There are two ways to get the binary on the box, and the choice is a fleet policy, not a preference.

Path A — MSI with the ENABLED_COLLECTORS property. The installer creates a service, registers the WMI provider, and is the only supported way to set collectors at install time.

# Run as Administrator
msiexec /i windows_exporter-0.27.1-amd64.msi ^
  ENABLED_COLLECTORS="cpu,cs,logical_disk,net,os,service,system,process" ^
  COLLECTOR_SERVICE_SERVICES_WHERECLAUSE="Name LIKE 'W3SVC'" ^
  /qn

sc.exe query windows_exporter
netstat -ano | findstr :9182

Path B — zip + NSSM for greenfield fleets. This is the path I use when the box is not domain-joined yet or when an MSI would trip a change-management review. Drop the binary, point a config at it, register it.

mkdir C:\exporter
Expand-Archive windows_exporter-0.27.1-amd64.zip -DestinationPath C:\exporter

# config.yml
cat > C:\exporter\config.yml <<'EOF'
collectors:
  enabled: cpu,cs,logical_disk,net,os,service,system,process
  service:
    services-where: "Name LIKE 'W3SVC'"
EOF

nssm install windows_exporter C:\exporter\windows_exporter.exe
nssm set windows_exporter AppParameters "--config C:\exporter\config.yml --telemetry.addr :9182"
nssm set windows_exporter AppDirectory C:\exporter
nssm set windows_exporter Start SERVICE_AUTO_START
nssm start windows_exporter

The MSI defaults to --telemetry.addr :9182 and a logical_disk,os,service,system collector set. If you install the MSI and never pass ENABLED_COLLECTORS, you get a thin, safe default. That is fine for a quick dashboard but you will miss network and CPU entirely.

One gotcha on both paths: the service account matters. wmi_exporter runs as LocalSystem by default, which can read everything. If your security baseline forces a low-privilege service account, you must grant it Win32_PerfFormattedData and Win32_Service WMI namespace read, or you will see windows_exporter_collector_duration_seconds errors and empty series. I hit exactly this; it is incident 5.2 below.

2. Collector Inventory: What Is On, What Needs Enabling, What Bites

The collector list is the single most misunderstood part of this stack. Here is the inventory I keep pinned in my runbook.

Collector	Default (MSI)	Perf profile	Notes
`cpu`	On	Low	Load average per core. Enable always.
`cs`	On	Low	System info, hostname, OS version. Cheap.
`logical_disk`	On	Medium	The big one for disk. Needs a volume filter (see 5.4).
`os`	On	Low	Memory, uptime, time. Enable always.
`service`	On	Medium	One series per service. Use `services-where` to trim.
`system`	On	Low	Context switches, threads, handles, processes.
`net`	Off	Medium	NIC bytes/packets. Turn on or you cannot alert on traffic.
`process`	Off	High	Per-process series. Cardinality bomb; see 5.5.
`iis`	Off	Medium	W3SVC only. Requires the IIS WMI provider and `services-where`.
`physical_disk`	Off	Medium	Per-physical-disk counters. Useful when logical_disk is too noisy.
`memory`	Off	Low	Use instead of `os` when you want the memory-specific counters.
`tcp`	Off	Low	TCP connection states by IPv4/IPv6. Good for connection exhaustion.

Two rules I enforce. First, net and memory are cheap and almost always worth enabling even though they are off by default — "network is not monitored" is a terrible first finding on an incident. Second, treat process and iis as opt-in after you have sized them, never as part of the default MSI install.

The cardinality math is the trap. process emits windows_process_handles, windows_process_threads, and a handful of process counters per process. A box running 200 processes becomes roughly 600-1000 new time series at every scrape. At a 15-second scrape interval that is tens of thousands of samples per minute per host. Multiply by 50 hosts and you have a Prometheus memory problem that has nothing to do with your application.

3. Windows Firewall: The Dual-Profile Trap

This is the number one reason "wmi_exporter is installed but Prometheus says target down." The MSI adds an inbound rule, but it is scoped to a specific network profile. A Windows host on a domain can simultaneously present a Domain profile to the LAN and a Private profile to a second NIC. If your Prometheus host reaches the exporter over the second NIC, the rule does not apply and the packet is dropped.

# This rule is Domain-profile only (the MSI default behavior)
New-NetFirewallRule -DisplayName "wmi_exporter 9182 Domain" `
  -Direction Inbound -Protocol TCP -LocalPort 9182 `
  -Profile Domain -Action Allow

# THIS is the fix for the dual-NIC / mixed-profile trap:
New-NetFirewallRule -DisplayName "wmi_exporter 9182 all profiles" `
  -Direction Inbound -Protocol TCP -LocalPort 9182 `
  -Profile Any -Action Allow

I also always verify with the legacy command because that is what the change ticket references:

netsh advfirewall firewall add rule name="wmi_exporter 9182" ^
  dir=in action=allow protocol=TCP localport=9182 profile=any

Then test from the Prometheus host, not from the server console:

curl -s http://10.0.1.24:9182/metrics | head -20
nc -vz 10.0.1.24 9182

If nc connects but curl returns empty, the exporter is up but the firewall is still eating it, or the exporter bound to 127.0.0.1. Check the bind address. The MSI binds to :9182 on all interfaces, but if you set --telemetry.addr 127.0.0.1:9182 to be safe, remote scrapes will never work. I have seen both.

4. Prometheus Scrape Config

A single job with per-host labels is the right shape for a fleet. Resist the urge to make one job per host.

# prometheus.yml fragment
scrape_configs:
  - job_name: windows
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - "10.0.1.24:9182"   # DC01
          - "10.0.1.25:9182"   # APP01
          - "10.0.1.26:9182"   # SQL01
        labels:
          env: prod
          os: windows
    relabel_configs:
      - source_labels: [__address__]
        regex: "([^:]+):9182"
        target_label: instance
        replacement: "$1"

Two tuning notes. First, scrape_timeout: 10s matters. Some logical_disk and process queries on a busy box can take longer than the default 10s timeout, which makes Prometheus mark the target as down even though the exporter is healthy. If you see target up but windows_exporter_collector_duration_seconds > 5, bump the timeout before you blame the network. Second, the instance relabel strips the port so your dashboards read DC01 instead of 10.0.1.24:9182.

5. Five Real Incidents

5.1 Port 9182 Unreachable — Dual Firewall Profile

Prometheus showed target down, but curl from a second RDP session on the same server returned a full metric dump. The exporter was fine; the problem was the firewall rule. The server had two NICs: the management LAN on the Domain profile and a storage network on the Private profile. Prometheus scraped over the storage network, so the Domain-only rule never applied. Fix: added a second profile=any rule (section 3) and documented that every new host needs both the Domain and the Any rule.

5.2 Service Account Lacked WMI Permission

After a security hardening pass, the windows_exporter service was switched from LocalSystem to a managed service account. Every collector started returning empty series and windows_exporter_collector_duration_seconds went to zero. WMI denied the account read on Win32_PerfFormattedData. Fix: granted the account Remote Access on the root\cimv2 namespace and added it to the Performance Monitor Users group. Never run the exporter as a truly unprivileged account without granting WMI read.

5.3 Windows Time Jump Caused a False `rate()` Alarm

At 03:00 an alert on rate(windows_cpu_time_total[5m]) > 0.9 fired for every host at once. Nothing was actually consuming CPU. The culprit was the Windows Time service applying a large forward correction, so the timestamp delta between samples collapsed and rate() divided by near-zero. Fix: added for: 5m to the alert and a windows_time_time_offset_seconds > 10 query to the dashboard. Rule: rate() on any counter is only trustworthy when the clock is stable.

5.4 Disk Queue False Positive From the Drive Letter Filter

windows_logical_disk_idle_seconds was alerting on a box with an idle disk. The problem: I had not filtered by volume. The logical_disk collector reports every mounted volume, including the hidden System Reserved partition and any Recovery volume. Summing across all of them made the queue appear saturated. Fix: always filter by drive letter.

# correct: only C:
1 - avg by (instance) (windows_logical_disk_idle_seconds{volume="C:"})
  > 0.9

5.5 Process Collector Blew Up Prometheus Memory

A 500-process terminal server was enabled with the process collector as part of the default set. Within a day Prometheus head memory doubled and query latency went from 300 ms to 4 s. The process collector had created thousands of series and every 15-second scrape was adding to the index. Fix: disabled process, enabled net, and left process_include empty so only a whitelist of critical processes is ever emitted.

collectors:
  enabled: cpu,cs,logical_disk,net,os,service,system
  process:
    process-include: "^(sqlservr|w3wp|lsass|svchost)$"

6. Key Metrics and Their PromQL

These are the eight queries I put on every Windows dashboard.

Metric	What it tells you	PromQL
CPU busy %	Utilization across cores	`100 - avg by (instance) (windows_cpu_time_total{mode="idle"} / 100) * 100`
Memory available	Headroom before paging	`windows_os_physical_memory_free_bytes`
Memory % used	Pressure ratio	`1 - windows_os_physical_memory_free_bytes / windows_os_physical_memory_total_bytes`
Disk queue length	I/O saturation	`windows_logical_disk_avg_disk_queue_length{volume="C:"}`
Disk busy %	Time the disk was active	`100 - windows_logical_disk_idle_seconds{volume="C:"} / 100`
Handle count	Leak indicator, process exhaustion	`sum(windows_system_handle_count)`
Process count	Zombie / runaway process check	`sum(windows_system_process_count)`
Network traffic	NIC throughput	`sum by (instance) (rate(windows_net_bytes_received_total[5m]))`

For handle count, a sustained linear climb over hours is a classic handle leak. windows_system_handle_count is a gauge, so you can't rate() it — plot it raw and look for a slope, not a spike.

7. Grafana Dashboard: The Panel Layout That Works

I build the dashboard from three rows. Row 1 is "load" (CPU, memory). Row 2 is "disk and IO" (queue, busy %, free space by volume). Row 3 is "process and network" (handles, processes, NIC traffic). The key is a instance template variable.

{
  "title": "Windows Server Overview",
  "templating": {
    "list": [
      {
        "name": "instance",
        "type": "query",
        "query": "label_values(windows_cpu_time_total, instance)"
      }
    ]
  },
  "panels": [
    {
      "title": "CPU busy %",
      "type": "timeseries",
      "targets": [{
        "expr": "100 - avg by (instance) (windows_cpu_time_total{mode=\"idle\"} / 100) * 100"
      }]
    },
    {
      "title": "Disk queue C:",
      "type": "timeseries",
      "targets": [{
        "expr": "windows_logical_disk_avg_disk_queue_length{volume=\"C:\", instance=~\"$instance\"}"
      }]
    }
  ]
}

The template variable makes every panel follow the host selector. Without it, you are either plotting all hosts on one chart (unreadable) or maintaining a dashboard per host (unmaintainable).

8. Alert Rules

groups:
  - name: windows
    rules:
      - alert: WindowsHighCpu
        expr: (100 - avg by (instance) (windows_cpu_time_total{mode="idle"} / 100) * 100) > 85
        for: 5m
        labels: { severity: warning }
        annotations: { summary: "CPU > 85% on {{ $labels.instance }}" }

      - alert: WindowsLowMemory
        expr: windows_os_physical_memory_free_bytes / windows_os_physical_memory_total_bytes < 0.1
        for: 10m
        labels: { severity: critical }
        annotations: { summary: "Free memory under 10% on {{ $labels.instance }}" }

      - alert: WindowsDiskQueueHigh
        expr: windows_logical_disk_avg_disk_queue_length{volume="C:"} > 5
        for: 5m
        labels: { severity: warning }
        annotations: { summary: "Disk queue on C: is saturated on {{ $labels.instance }}" }

      - alert: WindowsTimeSkew
        expr: abs(windows_time_time_offset_seconds) > 10
        for: 2m
        labels: { severity: warning }
        annotations: { summary: "Clock skew > 10s on {{ $labels.instance }}" }

      - alert: WindowsHandleLeak
        expr: windows_system_handle_count > 50000
        for: 30m
        labels: { severity: critical }
        annotations: { summary: "Handle count high on {{ $labels.instance }}" }

WindowsTimeSkew is the alert I recommend adding to every fleet. It catches the root cause of the 5.3 incident before it masquerades as a CPU or latency problem.

FAQ

Q: Why is wmi_exporter on :9182 unreachable from Prometheus but reachable locally?

A: The Windows Firewall inbound rule for port 9182 usually applies to the Domain profile only. If the Prometheus host is on a different network profile, the connection is dropped. Add a second rule scoped to the Private profile or apply it to all profiles.

Q: Why does my disk queue metric show a false alarm on the system drive?

A: Logical disk metrics carry a volume label like C: or D:. Without a drive letter filter, every mounted volume (including the hidden System Reserved one) is summed. Filter with windows_logical_disk_idle_seconds{volume="C:"}.

Q: Why did Prometheus memory spike after enabling the process collector?

A: The process collector emits a high-cardinality series per process per handle and thread. On a busy host that is thousands of series, and every scrape grows the in-memory index. Use the process collector with the process_include filter or disable it.

Q: Why did my rate() alert fire at 03:00 even though nothing changed?

A: Windows time sync jumped the clock forward, so the timestamp delta in the rate window became negative or collapsed to zero. Use for: 5m on the alert and check windows_time_time_offset_seconds before blaming the service.

Q: Can I run wmi_exporter without a Windows service or MSI install?

A: Yes. Unzip the binary and register it with NSSM as a service. This is the green-deploy path for fleets where you cannot run an MSI but can drop a binary and a config file.