Prometheus Recording Rules 实战：预计算指标提升查询性能与降低告警延迟

Q: Recording Rules 和 Alerting Rules 有什么区别？

Recording Rules 创建新指标（预计算），Alerting Rules 触发告警（条件判断）。两者 YAML 结构相同，但 record 字段 vs alert 字段。 通常 Recording Rules 为 Alerting Rules 服务——先预计算指标，再基于预计算结果写简洁的告警规则。

Q: 什么时候该用 Recording Rules？

三个信号：① 同一个复杂查询在多个 Dashboard 重复使用；② 查询耗时超过 1 秒影响 Dashboard 加载；③ 告警规则表达式过于复杂难以维护。 不要为简单查询创建 Recording Rule，反而会增加维护成本。

Q: Recording Rules 会影响查询实时性吗？

会，延迟等于 rule evaluation interval。建议 infra 层 30s、app 层 15s、business 层 60s。 如果业务要求秒级实时性，不要用 Recording Rules，直接用 PromQL 查询。

Q: 如何监控 Recording Rules 的性能？

四个关键指标：① `prometheus_rule_evaluation_duration_seconds` 看评估耗时； ② `prometheus_rule_evaluation_failures_total` 看失败率； ③ `prometheus_tsdb_head_series` 看新增 Time Series 数量； ④ `process_resident_memory_bytes` 看内存占用。 建议设置告警：评估耗时 P99 > 10s 或失败率 > 1%。

生产环境 Prometheus 运行半年后，Dashboard 加载变慢、告警延迟增加，90% 的原因是 PromQL 查询过于复杂。Recording Rules 是官方推荐的优化方案——把复杂查询预计算成新指标，Dashboard 和告警直接查预计算结果，性能提升 10 倍以上。

为什么需要 Recording Rules

没有 Recording Rules 的问题

# Dashboard 查询（每次实时计算）
expr: sum by (job) (rate(http_requests_total[5m])) / sum by (job) (rate(http_requests_total[5m])) * 100

# 告警规则（复杂表达式）
expr: |
  (sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))
   / sum by (instance) (node_cpu_seconds_total)) * 100 > 80

问题： - 每次查询都要实时计算，Dashboard 加载慢 - 复杂表达式难以维护，改一处影响多处 - 多 Dashboard 重复相同计算，浪费资源

Recording Rules 的解决方案

# 预计算指标
groups:
  - name: http_rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_request_duration_seconds:avg
        expr: avg by (job) (rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]))

# Dashboard 直接查预计算指标
expr: job:http_requests:rate5m

好处： - Dashboard 查询速度提升 10x+ - 告警规则简洁易维护 - 重复计算只执行一次

Recording Rules 命名规范

官方推荐命名格式：level:metric:operations

部分	说明	示例
`level`	聚合维度	`instance`、`job`、`cluster`
`metric`	原始指标名（去掉 `_total`）	`http_requests`、`node_cpu`
`operations`	操作类型和时间窗口	`rate5m`、`avg1h`、`sum`

正确示例：

# 实例级别：CPU 使用率 5 分钟平均
- record: instance:node_cpu_utilization:avg5m
  expr: avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

# Job 级别：请求速率
- record: job:http_requests:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))

# 集群级别：错误率
- record: cluster:http_error_rate:ratio_rate5m
  expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

错误示例：

# ❌ 命名不清晰
- record: cpu_usage
  expr: ...

# ❌ 缺少操作类型
- record: instance_http_requests
  expr: ...

# ❌ 高基数标签（会导致 Time Series 爆炸）
- record: user:http_requests:rate5m
  expr: sum by (user) (rate(http_requests_total[5m]))

分层设计原则

生产环境建议三层设计：

第一层：Raw Metric Normalization（原始指标规范化）

groups:
  - name: raw_normalization
    interval: 15s    # 高频层，15 秒评估
    rules:
      # CPU 使用率
      - record: instance:node_cpu_utilization:rate5m
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

      # 内存使用率
      - record: instance:node_memory_utilization:ratio
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

      # 磁盘 I/O
      - record: instance:node_disk_io_time:rate5m
        expr: rate(node_disk_io_time_seconds_total[5m])

第二层：Infrastructure Aggregation（基础设施聚合）

groups:
  - name: infra_aggregation
    interval: 30s    # 中频层，30 秒评估
    rules:
      # Job 级别聚合
      - record: job:node_cpu_utilization:avg
        expr: avg by (job) (instance:node_cpu_utilization:rate5m)

      # 集群级别聚合
      - record: cluster:node_cpu_utilization:p99
        expr: histogram_quantile(0.99, sum by (le, cluster) (rate(node_cpu_seconds_total[5m])))

第三层：Business Metrics（业务指标）

groups:
  - name: business_metrics
    interval: 60s    # 低频层，60 秒评估
    rules:
      # 订单成功率
      - record: business:order_success_rate:ratio_rate5m
        expr: sum(rate(order_success_total[5m])) / sum(rate(order_total[5m]))

      # API 响应时间 P99
      - record: business:api_latency:p99_5m
        expr: histogram_quantile(0.99, sum by (le, service) (rate(api_request_duration_seconds_bucket[5m])))

为什么分层： - 不同层级数据新鲜度要求不同 - 高频层频繁计算，低频层可以延迟 - 便于单独调优和故障隔离

性能优化技巧

1. 避免高基数标签

# ❌ 危险：user_id 是高基数标签，会导致 Time Series 爆炸
- record: user:http_requests:rate5m
  expr: sum by (user_id) (rate(http_requests_total[5m]))

# ✅ 安全：用 topk 限制数量
- record: top_users:http_requests:rate5m
  expr: topk(10, sum by (user_id) (rate(http_requests_total[5m])))

2. 合理设置 interval

groups:
  - name: critical_metrics
    interval: 15s    # 关键指标高频计算
    rules:
      - record: instance:cpu:utilization
        expr: ...

  - name: business_metrics
    interval: 60s    # 业务指标低频计算
    rules:
      - record: business:revenue:daily
        expr: ...

3. 使用 without() 保留关键标签

# ❌ 丢失 job 标签，告警时无法定位
- record: http_requests:rate5m
  expr: sum(rate(http_requests_total[5m]))

# ✅ 保留 job 标签
- record: job:http_requests:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))

4. 链式规则优化

# 第一层：预计算基础指标
- record: instance_path:http_requests:rate5m
  expr: sum by (instance, path) (rate(http_requests_total[5m]))

# 第二层：基于第一层聚合（利用缓存结果）
- record: path:http_requests:rate5m
  expr: sum by (path) (instance_path:http_requests:rate5m)

完整生产模板

# recording_rules.yml
groups:
  # 第一层：Raw Metric Normalization
  - name: raw_normalization
    interval: 15s
    rules:
      # CPU
      - record: instance:node_cpu_utilization:rate5m
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

      # Memory
      - record: instance:node_memory_utilization:ratio
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

      # Disk I/O
      - record: instance:node_disk_utilization:rate5m
        expr: rate(node_disk_io_time_seconds_total[5m])

      # Network
      - record: instance:node_network_transmit_bytes:rate5m
        expr: rate(node_network_transmit_bytes_total[5m])

      # HTTP Requests
      - record: instance:http_requests:rate5m
        expr: sum by (instance) (rate(http_requests_total[5m]))

      # HTTP Errors
      - record: instance:http_errors:rate5m
        expr: sum by (instance) (rate(http_requests_total{status=~"5.."}[5m]))

  # 第二层：Infrastructure Aggregation
  - name: infra_aggregation
    interval: 30s
    rules:
      # Job 级别 CPU
      - record: job:node_cpu_utilization:avg
        expr: avg by (job) (instance:node_cpu_utilization:rate5m)

      # Job 级别内存
      - record: job:node_memory_utilization:avg
        expr: avg by (job) (instance:node_memory_utilization:ratio)

      # 集群级别错误率
      - record: cluster:http_error_rate:ratio_rate5m
        expr: |
          sum(rate(http_errors:rate5m[5m]))
          /
          sum(rate(http_requests:rate5m[5m]))

  # 第三层：Business Metrics
  - name: business_metrics
    interval: 60s
    rules:
      # 订单成功率
      - record: business:order_success_rate:ratio_rate5m
        expr: |
          sum(rate(order_success_total[5m]))
          /
          sum(rate(order_total[5m]))

      # API 响应时间 P99
      - record: business:api_latency:p99_5m
        expr: histogram_quantile(0.99, sum by (le, service) (rate(api_request_duration_seconds_bucket[5m])))

      # 用户活跃度
      - record: business:active_users:count_5m
        expr: count by (service) (increase(user_login_total[5m]))

性能监控告警

# 监控 Recording Rules 本身的健康
groups:
  - name: recording_rules_health
    rules:
      # 评估耗时告警
      - alert: RecordingRuleEvaluationSlow
        expr: prometheus_rule_evaluation_duration_seconds{quantile="0.99"} > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Recording rule evaluation slow"
          description: "Rule {{ $labels.rule }} evaluation P99 is {{ $value }}s"

      # 评估失败告警
      - alert: RecordingRuleEvaluationFailures
        expr: rate(prometheus_rule_evaluation_failures_total[5m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Recording rule evaluation failures"
          description: "Rule {{ $labels.rule }} has {{ $value }} failures/sec"

      # Time Series 数量告警
      - alert: HighRecordingRuleSeriesCount
        expr: prometheus_tsdb_head_series > 1000000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Too many Time Series from recording rules"
          description: "Current series count: {{ $value }}"

翻车案例复盘

案例 1：Time Series 爆炸

现象：Prometheus 内存使用从 4GB 涨到 32GB，查询变慢。

根因：创建了过多 Recording Rules，且使用了高基数标签（如 user_id、request_id）。

修复：

# ❌ 删除高基数规则
# - record: user:http_requests:rate5m
#   expr: sum by (user_id) (rate(http_requests_total[5m]))

# ✅ 改用 topk 限制
- record: top_users:http_requests:rate5m
  expr: topk(100, sum by (user_id) (rate(http_requests_total[5m])))

案例 2：链式规则循环依赖

现象：Prometheus 启动失败，报错 rule group circular dependency detected。

根因：

groups:
  - rules:
      - record: metric_a
        expr: metric_b + 1     # 依赖 metric_b

      - record: metric_b
        expr: metric_a + 1     # 依赖 metric_a ← 循环！

修复：拆分成两个 group，确保单向依赖。

案例 3：interval 设置不当导致数据延迟

现象：Dashboard 显示数据延迟 5 分钟。

根因：interval: 300s 设置过大。

修复：根据业务需求调整 interval，关键指标建议 15-30s。

写在最后

Recording Rules 是 Prometheus 性能优化的核心工具，但要用对地方：

只在必要时使用——简单查询不需要预计算
遵循命名规范——便于维护和调试
分层设计——infra 层高频，business 层低频
监控规则健康——评估耗时、失败率、Series 数量

记住：Recording Rules 是 trade-off——用存储空间换查询速度。合理控制 Series 数量，别让预计算变成新的性能瓶颈。