AI 模型评估与基准测试 · 运维手册
适用场景：评估和比较不同 LLM 模型，建立持续的质量监控体系。
1. 评估指标体系
1.1 核心指标
| 类别 | 指标 | 计算方法 | 目标值 |
|---|---|---|---|
| 准确性 | Accuracy | 正确回答数/总数 | > 85% |
| F1 Score | 2×Precision×Recall/(Precision+Recall) | > 0.8 | |
| 生成质量 | ROUGE-L | 最长公共子序列召回率 | > 0.6 |
| BLEU | n-gram 精确度 | > 0.4 | |
| BERTScore | 语义相似度 | > 0.85 | |
| 效率 | P50 Latency | 50% 请求延迟 | < 1s |
| P99 Latency | 99% 请求延迟 | < 3s | |
| Token/sec | 生成速度 | > 50 | |
| 成本 | Cost/Request | 单次请求成本 | < $0.01 |
| Cost/1K Tokens | 千 Token 成本 | 最低 | |
| 稳定性 | Error Rate | 失败请求比例 | < 1% |
| Availability | 服务可用性 | > 99.9% |
1.2 评估框架
评估流程：
1. 定义评估目标（场景、指标、阈值）
2. 构建测试集（标注数据）
3. 运行模型评估（批量推理）
4. 分析结果（指标对比）
5. 决策（选择/调优/回滚）
6. 持续监控（定期回归测试）
2. 基准测试
2.1 MTEB 基准
MTEB（Massive Text Embedding Benchmark）是 Embedding 模型评估标准：
from mteb import MTEB
# 选择任务
evaluation = MTEB(tasks=[
"STS14", # 语义相似度
"Summarization", # 摘要
"Instruction Retrieval" # 指令检索
])
# 运行评估
results = evaluation.run(model)
print(results)
2.2 SuperCLUE 中文基准
from superclue import SuperCLUE
# 初始化
benchmark = SuperCLUE()
# 运行测试
results = benchmark.evaluate(
model="Qwen/Qwen2.5-7B-Instruct",
tasks=["coding", "math", "reasoning", "knowledge"]
)
# 查看排名
benchmark.show_leaderboard()
2.3 自定义运维基准
OPS_BENCHMARK = [
{
"category": "network",
"question": "Nginx 502 错误可能的原因有哪些？",
"expected_keywords": ["upstream", "connection", "timeout"],
"difficulty": "medium"
},
{
"category": "kubernetes",
"question": "Pod 处于 CrashLoopBackOff 状态如何排查？",
"expected_keywords": ["logs", "restart", "oom"],
"difficulty": "hard"
},
{
"category": "monitoring",
"question": "Prometheus 告警如何配置静默规则？",
"expected_keywords": ["silence", "alertmanager", "match"],
"difficulty": "medium"
}
]
3. 测试集构建
3.1 数据来源
| 来源 | 优点 | 缺点 | 占比建议 |
|---|---|---|---|
| 历史工单 | 真实场景 | 可能过时 | 40% |
| 知识库文档 | 覆盖全面 | 缺乏难度 | 30% |
| 专家标注 | 质量高 | 成本高 | 20% |
| 合成数据 | 可批量生成 | 可能不真实 | 10% |
3.2 分层抽样
import random
def stratified_sample(dataset: list, labels: dict) -> list:
"""分层抽样确保多样性"""
sampled = []
for category, count in labels.items():
# 按难度分层
easy = [d for d in dataset if d['category'] == category and d['difficulty'] == 'easy']
medium = [d for d in dataset if d['category'] == category and d['difficulty'] == 'medium']
hard = [d for d in dataset if d['category'] == category and d['difficulty'] == 'hard']
# 按比例抽样
sampled.extend(random.sample(easy, min(5, len(easy))))
sampled.extend(random.sample(medium, min(10, len(medium))))
sampled.extend(random.sample(hard, min(5, len(hard))))
return sampled
3.3 质量检查
def validate_dataset(dataset: list) -> dict:
"""测试集质量检查"""
checks = {
"total": len(dataset),
"categories": len(set(d['category'] for d in dataset)),
"difficulty_dist": {
"easy": sum(1 for d in dataset if d['difficulty'] == 'easy'),
"medium": sum(1 for d in dataset if d['difficulty'] == 'medium'),
"hard": sum(1 for d in dataset if d['difficulty'] == 'hard')
},
"avg_length": sum(len(d['question']) for d in dataset) / len(dataset)
}
# 检查重复
questions = [d['question'] for d in dataset]
checks["duplicate_rate"] = 1 - len(set(questions)) / len(questions)
return checks
4. A/B 测试
4.1 实验设计
class ABTest:
def __init__(self, name: str):
self.name = name
self.variants = {}
self.metrics = {}
def add_variant(self, variant_id: str, model: str):
"""添加实验组"""
self.variants[variant_id] = model
def assign_user(self, user_id: str) -> str:
"""用户分流（Hash 确保一致性）"""
hash_val = hash(user_id) % 100
if hash_val < 50:
return "control"
else:
return "treatment"
def record_result(self, variant: str, result: dict):
"""记录结果"""
if variant not in self.metrics:
self.metrics[variant] = {"scores": [], "latencies": []}
self.metrics[variant]["scores"].append(result["score"])
self.metrics[variant]["latencies"].append(result["latency"])
4.2 统计分析
from scipy import stats
def analyze_ab_test(control_scores: list, treatment_scores: list) -> dict:
"""A/B 测试结果分析"""
# T 检验
t_stat, p_value = stats.ttest_ind(control_scores, treatment_scores)
# 效应量（Cohen's d）
pooled_std = np.sqrt((np.var(control_scores) + np.var(treatment_scores)) / 2)
cohens_d = (np.mean(treatment_scores) - np.mean(control_scores)) / pooled_std
return {
"p_value": p_value,
"significant": p_value < 0.05,
"effect_size": cohens_d,
"control_mean": np.mean(control_scores),
"treatment_mean": np.mean(treatment_scores)
}
5. 持续监控
5.1 质量 Dashboard
# 每日自动评估
def daily_quality_check():
"""每日质量检查"""
test_cases = load_test_cases()
results = []
for case in test_cases:
response = call_model(case["question"])
score = evaluate_response(case, response)
results.append({
"case_id": case["id"],
"score": score,
"latency": response["latency"],
"model": response["model"]
})
# 记录指标
save_metrics(results)
# 检查是否触发告警
avg_score = np.mean([r["score"] for r in results])
if avg_score < 0.7:
send_alert(f"模型质量下降：平均分 {avg_score:.2f}")
5.2 趋势分析
import pandas as pd
def analyze_trend(metrics: pd.DataFrame, window: int = 7) -> dict:
"""趋势分析"""
# 移动平均
metrics["moving_avg"] = metrics["score"].rolling(window=window).mean()
# 检测异常
threshold = metrics["moving_avg"].mean() - 2 * metrics["moving_avg"].std()
anomalies = metrics[metrics["score"] < threshold]
return {
"trend": "improving" if metrics["moving_avg"].iloc[-1] > metrics["moving_avg"].iloc[0] else "declining",
"anomalies": len(anomalies),
"current_score": metrics["score"].iloc[-1]
}
6. 评估报告模板
# LLM 模型评估报告
## 测试概览
- 测试时间：2026-10-23
- 测试用例：200 条
- 模型版本：Qwen2.5-7B-Instruct-v2
## 核心指标
| 指标 | 当前值 | 目标值 | 状态 |
|---|---|---|---|
| Accuracy | 87% | 85% | ✅ |
| P99 Latency | 2.1s | 3s | ✅ |
| Cost/1K Tokens | $0.001 | $0.002 | ✅ |
| User Satisfaction | 4.2/5 | 4.0 | ✅ |
## 问题清单
1. [P2] 复杂推理题得分偏低（65% vs 目标 80%）
2. [P3] 长文档处理超时率 3%
## 建议
- 下周：针对复杂推理优化 Prompt
- 下月：考虑升级到 70B 模型