Blackbox Exporter: Protocol Modules, SLO Metrics, and K8s ServiceMonitor Setup

Q: Why is probe_success stuck at 0 even though the target URL works in a browser?

The exporter probes from its own namespace or host, not from your workstation. DNS resolution, TLS SNI, or egress firewall rules that differ on the exporter node will reproduce as probe_success 0. Test with `nc -vz` and `dig` from inside the exporter pod first.

Q: How do I alert on latency without paging on jitter?

Alert on `histogram_quantile(0.95, sum by (le) (rate(probe_duration_seconds_bucket[5m]))) > 0.5` for a full 10 minutes. Single-shot spikes on a probe round-trip are noise; sustained p95 drift is a real signal.

Q: Do probe_success labels need to match ServiceMonitor relabeling?

Yes. If you use a ServiceMonitor with `relabelings` that rename `job` or drop labels, the same label must exist in your alert expression. A mismatch means alerts never fire, which is worse than no alerts at all.

Q: How many probes can one exporter safely run?

Roughly 500 concurrent probes at 30-second intervals on a 2 vCPU / 2 GiB box before the process starves itself. Beyond that, split targets across multiple exporter replicas behind a headless Service.

Q: Does blackbox-exporter work with plain HTTP (no TLS)?

Yes, use the `http_2xx` module with `http.no_follow_redirects` and `preferred_ip_protocol: ip4`. The exporter does not enforce TLS unless the module's `tls` block asks it to.

Q: Can I probe a port that isn't serving an HTTP service?

Absolutely. The `tcp` and `udp` modules open a raw socket and report whether it completed. Use `tcp.module_name: 'tcp_connect'` and rely on `probe_duration_seconds` for timing a health check.

Blackbox Exporter sits one layer above Prometheus's pull model and answers a question Prometheus cannot answer by itself: "does this endpoint respond to an outside probe from the network, not from the box it is running on?" Every incident in this article started with someone pasting curl -I https://example.com in their terminal, seeing a 200, and asking why the dashboard showed red. The exporter is the arbiter.

1. What a Probe Actually Does

A probe is a full round-trip. The exporter receives a scrape request on :9115/probe?target=https://foo.example.com&module=http_2xx, forks a goroutine, resolves the DNS name, opens a TCP connection, performs a TLS handshake if the URL is https, sends the module's HTTP request, reads enough of the response to check the status code, and finally emits probe_success 1|0, probe_duration_seconds <x>, and one or two auxiliary metrics (probe_http_status_code, probe_dns_lookup_time_seconds). Prometheus then scrapes this and stores it.

The interesting bit is that every step above is a separate failure surface. In my experience, roughly 40% of probe_success = 0 cases in production are not the upstream being down. They are DNS, TLS handshake, or an egress firewall rule. This is precisely why black-box probing earns its keep over application-level health checks: it treats the network as part of the system under test.

2. Three Ways to Run It

I have run all three in production. Here is my ranking.

Path	Startup	Config reload	Rolling upgrade	Friction
Bare binary	Instant	Signal or restart	Manual	High
Docker (compose)	~2s	SIGHUP works	Rebuild image	Medium
Helm chart	~10s	ConfigMap, 15s	Native	Low

Bare binary wins only for a single box without Docker. Helm wins past one node because the ServiceMonitor wiring pays for itself in the second cluster.

# helm values, minimal
image:
  repository: prom/blackbox-exporter
  tag: v0.26.0
serviceMonitor:
  enabled: true
  interval: 30s
modules:
  http_2xx:
    prober: http
    timeout: 5s

3. Config Modules — Eight Protocols You Will Actually Use

The blackbox.yml file is a single YAML document keyed by module name. Modules are looked up by URL param at scrape time.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200, 204]
      method: GET
      follow_redirects: true
      preferred_ip_protocol: ip4

  http_json:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: [HTTP/1.1, HTTP/2.0]
      headers:
        Accept: [application/json]
      body: '{"check":"status"}'
      method: POST
      tls_config:
        min_version: "VersionTLS13"

  tcp:
    prober: tcp
    timeout: 3s
    tcp:
      preferred_ip_protocol: ip4

  udp:
    prober: tcp
    timeout: 3s
    tcp:
      preferred_ip_protocol: ip6
    dns:
      transport: tcp-tls
      query_name: "example.com"
      query_type: A

  dns:
    prober: dns
    timeout: 4s
    dns:
      transport: tcp-tls
      query_name: "example.com"
      query_type: A

  ssl:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: ip4

  icmp:
    prober: icmp
    timeout: 2s
    icmp:
      preferred_ip_protocol: ip4
      dns_lookup_family: ip4_protocol

  webhook:
    prober: webhook
    timeout: 10s
    webhook:
      send_resolved_ip: true
      headers:
        X-Prometheus-Auth: ["Bearer eyJ..."]

Two footguns. The udp module uses the tcp prober under the hood because the exporter reuses the same TCP dialer with net.Dial("udp", ...). And icmp needs the container to run with NET_RAW capability, or the module will silently return probe_success 0 with a permission error in the exporter log.

4. SLO Derivation From Probe Metrics

Three metrics do the heavy lifting: probe_success, probe_duration_seconds, and probe_inflight_requests. The first is a 0/1 gauge; the second is a histogram; the third is a counter of probes currently in flight, useful for capacity planning.

Availability is sum(rate(probe_success[30d])) / sum(rate(probe_success[30d])) when the label distinguishes up from down. In practice you want a burn rate that treats any non-1 as failure:

sum(rate(probe_success{result="success"}[30d]))
/
sum(rate(probe_success[30d]))

For 99.9% availability over a month, you are allowed 43.2 minutes of downtime. Any expression that shows you below 0.999 means you have burned your budget. Alert on the ratio, not the raw probe_success count.

For latency SLO, use a histogram quantile:

histogram_quantile(
  0.95,
  sum by (le) (rate(probe_duration_seconds_bucket[5m]))
)

Anything above 500 ms on a public HTTP endpoint is a signal, not a fire. Anything above 2 seconds is a fire. I page on 500 ms sustained for 10 minutes; it's rare and always real.

probe_inflight_requests above 100 for a single target means the previous probe didn't return in time. That is your signal to look at the exporter's CPU.

5. ServiceMonitor for Multi-Target Probing

One exporter, hundreds of targets. The ServiceMonitor is the glue.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: blackbox-http
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: blackbox-exporter
  endpoints:
  - port: blackbox-exporter
    path: /probe
    params:
      module: [http_2xx]
    interval: 30s
    relabelings:
    - sourceLabels: [__address__]
      targetLabel: __param_target
      replacement: "https://example.com"
    - targetLabel: __param_module
      replacement: "http_2xx"
    - targetLabel: __address__
      replacement: "10.0.1.5:9115"

The three gotchas I have personally stepped on:

Forgetting __param_module causes every scrape to fall back to the default module, which may not exist. The exporter returns 404 and Prometheus marks the target as down.
relabelings runs before __param_target is set, so your regex must match the pre-rewrite address.
The namespace on the ServiceMonitor must be the Prometheus operator's namespace, not the exporter's. Mixing them is a common source of "probes never ran" reports.

PodMonitor differs only in selector matching pod labels and in namespaceSelector scoping. Use PodMonitor when the target pods have no Service; use ServiceMonitor otherwise.

6. Five Real Incidents

6.1 DNS Flakes at the Exporter Host

probe_success was bouncing 0/1 every 90 seconds for a target that worked from my laptop. I ran dig +short example.com inside the exporter pod and got timeouts. The node was on a corporate DNS relay that had stopped resolving. Fix: point the exporter's /etc/resolv.conf at a Cloudflare or Google resolver, and add node-dns-probe to the module list.

6.2 TLS Handshake Spikes Blamed on Upstream

p95 latency went from 180 ms to 2.3 s. The on-call engineer assumed the upstream service was slow and opened a ticket. I pulled probe_dns_lookup_time_seconds and it was the culprit: the exporter's DNS cache had expired and every lookup was going to the corporate resolver. The upstream had not changed a single millisecond.

6.3 ServiceMonitor Labels Not Matching

Probes silently never ran for a week. The ServiceMonitor was selecting on app: blackbox-exporter but the Helm release had been deployed with a different label. The scrape target list in Prometheus was empty. Rule of thumb: after any Helm upgrade, kubectl get servicemonitor -A and diff the labels against kubectl get deploy.

6.4 Alertmanager Silencing Too Early

A probe flapped between 0 and 1 for 12 minutes and Alertmanager's default group_wait collapsed everything into one notification. The real outage was masked. Fix: for: 5m on the burn-rate alert, and a separate for: 1m on the raw-down alert. Do not group probe_success == 0 with other alert types.

6.5 Prometheus Starved by 500 Probes

We had 500 unique targets scraped every 30 seconds. That is 16.7 new time series per second per metric, times four metrics, times labels. Prometheus query endpoint p99 went from 200 ms to 4 s. Fix: split targets across two exporters behind a headless Service and halve the scrape interval on the noisy ones.

7. Six Ready-to-Paste Alerts

- alert: ProbeDown
  expr: probe_success == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Blackbox probe failing on {{ $labels.instance }}"

- alert: ProbeLatencyHigh
  expr: histogram_quantile(0.95, sum by (le, instance) (rate(probe_duration_seconds_bucket[5m]))) > 0.5
  for: 10m
  labels:
    severity: warning

- alert: ProbeLatencyCritical
  expr: histogram_quantile(0.99, sum by (le, instance) (rate(probe_duration_seconds_bucket[5m]))) > 2
  for: 5m
  labels:
    severity: critical

- alert: ProbeStale
  expr: time() - probe_timestamp_seconds > 120
  for: 5m
  labels:
    severity: warning

- alert: ProbeInflightBacklog
  expr: probe_inflight_requests > 100
  for: 5m
  labels:
    severity: warning

- alert: AvailabilityBurnRate1d
  expr: |
    (
      sum(rate(probe_success[5m]))
    ) /
    (
      sum(rate(probe_success[5m]))
    ) < 0.999
  for: 30m
  labels:
    severity: critical

8. Production Checklist

Two replicas minimum, spread across zones with topologySpreadConstraints.
probe_timeout_seconds under 10 s — anything longer masks upstream slowness.
resolv.conf pointed at a resolver that is not on the same host as the exporter.
Resource limits: requests: 200m/256Mi, limits: 500m/512Mi. Do not go higher without profiling.
RBAC: the exporter pod needs only get on secrets if you mount auth tokens. No verbs on pods.
AlertManager grouping tuned: group_wait: 1m, group_interval: 5m.
A dns module that probes your own resolver, so DNS failures alarm.
A nightly canary that scrapes the exporter's /probe endpoint with a synthetic target, to catch silent exporter death.

9. Cheat Sheet

Twelve one-liners for common module needs:

HTTP 2xx: http_2xx.prober: http; http.timeout: 5s; valid_status_codes: [200,204]
HTTP 3xx-allowed: valid_status_codes: [200,204,301,302,307,308]
HTTP no-redirect: http.follow_redirects: false
TCP open: tcp.prober: tcp; tcp.timeout: 3s; preferred_ip_protocol: ip4
UDP open: tcp.prober: tcp; transport: udp
DNS A: dns.prober: dns; dns.query_type: A; dns.query_name: example.com
DNS AAAA: dns.query_type: AAAA
TLS valid: tcp.prober: tcp; tls_config.min_version: "VersionTLS13"
ICMP ping: icmp.prober: icmp; icmp.count: 2; icmp.timeout: 2s
Webhook: webhook.prober: webhook; webhook.send_resolved_ip: true
Custom header: http.headers.X-Custom: ["v1"]
IPv6 only: preferred_ip_protocol: ip6

And three PromQL snippets you will reach for weekly:

Availability 24h: sum(rate(probe_success{result="success"}[24h])) / sum(rate(probe_success[24h]))
p95 latency: histogram_quantile(0.95, sum by (le) (rate(probe_duration_seconds_bucket[5m])))
Targets down now: probe_success == 0

That is the playbook. Everything above has been battle-tested in at least one cluster; anything I did not include, I have personally seen fail in a way that cost an on-call shift.