LLM Evaluation — Benchmarking and Quality Assurance

Use Case: Establish rigorous evaluation frameworks for LLM applications: benchmarks, testing, and continuous quality monitoring.

1. Evaluation Framework

Three-Layer Evaluation

Layer	Purpose	Method	Frequency
Baseline	Model capability	MTEB, standard benchmarks	Initial + updates
Task-specific	Domain performance	Custom test sets	Every deployment
Production	Real-world quality	User feedback, logs	Continuous

2. Standard Benchmarks (MTEB)

Popular MTEB Tasks

from mteb import MTEB

# Define evaluation tasks
tasks = [
    "STS14",           # Semantic textual similarity
    "Summarization",   # Text summarization quality
    "QuestionAnswering",  # QA accuracy
    "Retrieval",       # Information retrieval
    "Classification",  # Text classification
]

# Run evaluation
evaluation = MTEB(tasks=tasks)
results = evaluation.run(model, output_folder="./results")

Benchmark Results Interpretation

Benchmark	Good Score	Excellent Score	Use Case
MTEB STS	75+	85+	Semantic similarity
MTEB Summarization	ROUGE-1: 30+	ROUGE-1: 40+	Text summarization
MTEB Classification	Accuracy: 80%	Accuracy: 90%	Categorization
MMLU	65+	75+	General knowledge

3. Custom Test Sets

Test Set Structure

{
  "test_set": "customer_support_v1",
  "samples": [
    {
      "id": "cs_001",
      "input": "My order hasn't arrived after 2 weeks",
      "expected_output": "I understand your concern. Let me look up your order...",
      "expected_entities": {"order_status": "delayed"},
      "expected_sentiment": "empathetic",
      "difficulty": "standard"
    },
    {
      "id": "cs_002", 
      "input": "How do I reset my password?",
      "expected_output": "To reset your password...",
      "expected_entities": {"action": "password_reset"},
      "difficulty": "simple"
    }
  ],
  "metadata": {
    "created": "2026-10-23",
    "owner": "ops-team",
    "version": "1.0"
  }
}

Test Runner

class TestRunner:
    def __init__(self, test_set: list[dict]):
        self.tests = test_set
        self.results = []

    async def run_all(self, llm_client) -> dict:
        """Run all tests and collect results."""
        for test in self.tests:
            result = await self._run_test(test, llm_client)
            self.results.append(result)

        return self._generate_report()

    async def _run_test(self, test: dict, client) -> dict:
        """Run single test case."""
        start_time = time.time()

        try:
            response = await client.generate(test["input"])
            latency = time.time() - start_time

            return {
                "id": test["id"],
                "input": test["input"],
                "output": response,
                "expected": test["expected_output"],
                "latency_ms": latency * 1000,
                "passed": self._evaluate_response(test, response)
            }
        except Exception as e:
            return {
                "id": test["id"],
                "error": str(e),
                "passed": False
            }

    def _evaluate_response(self, test: dict, response: str) -> bool:
        """Evaluate response against expected output."""
        # Use LLM-as-judge for nuanced evaluation
        judge_prompt = f"""
        Evaluate if this response meets the requirements:

        Question: {test['input']}
        Expected: {test['expected_output']}
        Actual: {response}

        Does the actual response satisfy the expected criteria?
        Return JSON: {{"passed": true/false, "score": 0-1, "reason": "..."}}
        """

        judge_result = self._call_judge(judge_prompt)
        return judge_result["passed"]

4. A/B Testing Framework

Experimental Design

class ABTest:
    def __init__(self, name: str, variants: list[str]):
        self.name = name
        self.variants = variants
        self.traffic_split = self._calculate_split(variants)

    def _calculate_split(self, variants: list[str]) -> dict:
        """Calculate traffic distribution."""
        n = len(variants)
        if n == 1:
            return {variants[0]: 1.0}
        return {v: 1.0/n for v in variants}

    async def run_experiment(self, test_cases: list[dict]) -> dict:
        """Run A/B test across variants."""
        results = {v: {"inputs": [], "outputs": []} for v in self.variants}

        for test_case in test_cases:
            # Assign to variant based on hash
            variant = self._assign_variant(test_case["id"])
            output = await self._call_variant(variant, test_case["input"])
            results[variant]["inputs"].append(test_case["input"])
            results[variant]["outputs"].append(output)

        return self._analyze_results(results)

    def _assign_variant(self, test_id: str) -> str:
        """Deterministic variant assignment."""
        hash_val = hash(test_id) % 100
        cumulative = 0
        for variant, split in self.traffic_split.items():
            cumulative += split * 100
            if hash_val < cumulative:
                return variant
        return self.variants[-1]

Statistical Significance

from scipy import stats

def calculate_significance(control_results: list, variant_results: list) -> dict:
    """Calculate statistical significance of A/B test."""
    # T-test for continuous metrics
    t_stat, p_value = stats.ttest_ind(
        control_results, 
        variant_results
    )

    # Effect size (Cohen's d)
    pooled_std = stats.pooled_stdev([control_results, variant_results])
    effect_size = (
        sum(variant_results) / len(variant_results) - 
        sum(control_results) / len(control_results)
    ) / pooled_std

    return {
        "p_value": p_value,
        "significant": p_value < 0.05,
        "effect_size": effect_size,
        "interpretation": self._interpret_effect(effect_size)
    }

5. LLM-as-Judge

Evaluation Prompt Template

JUDGE_PROMPT = """
You are an expert evaluator assessing LLM output quality.

**Input**: {input}
**Expected Output**: {expected}
**Actual Output**: {actual}

**Evaluation Criteria**:
1. Correctness: Does the output accurately address the input?
2. Completeness: Does the output cover all required aspects?
3. Clarity: Is the output clear and well-structured?
4. Safety: Is the output free from harmful content?

**Scoring**:
- Correctness: 0-1 (1 = fully correct)
- Completeness: 0-1 (1 = covers all aspects)
- Clarity: 0-1 (1 = very clear)
- Safety: 0 or 1 (1 = safe)

**Overall Score**: Average of all criteria

Return JSON:
{{
  "correctness": 0.9,
  "completeness": 0.8,
  "clarity": 0.95,
  "safety": 1.0,
  "overall": 0.91,
  "verdict": "pass|fail",
  "notes": "Brief explanation"
}}
"""

Batch Evaluation

async def batch_evaluate(
    test_cases: list[dict],
    llm_client,
    judge_client
) -> list[dict]:
    """Evaluate multiple test cases in batch."""
    evaluations = []

    for i, test_case in enumerate(test_cases):
        # Generate response
        response = await llm_client.generate(test_case["input"])

        # Evaluate with judge
        judge_result = await judge_client.generate(
            JUDGE_PROMPT.format(
                input=test_case["input"],
                expected=test_case["expected"],
                actual=response
            )
        )

        evaluations.append({
            "test_id": test_case["id"],
            "response": response,
            "evaluation": parse_json(judge_result),
            "timestamp": datetime.utcnow().isoformat()
        })

        # Progress logging
        if (i + 1) % 10 == 0:
            print(f"Evaluated {i + 1}/{len(test_cases)}")

    return evaluations

6. Continuous Monitoring

Quality Drift Detection

class QualityMonitor:
    def __init__(self, baseline_scores: dict):
        self.baseline = baseline_scores
        self.recent_scores = deque(maxlen=1000)

    def update(self, score: float):
        """Add new quality score."""
        self.recent_scores.append(score)

    def check_drift(self) -> dict:
        """Detect quality drift from baseline."""
        if len(self.recent_scores) < 100:
            return {"status": "insufficient_data"}

        recent_avg = sum(self.recent_scores) / len(self.recent_scores)
        baseline_avg = self.baseline["overall"]

        drift = recent_avg - baseline_avg
        significance = self._check_significance(drift)

        return {
            "recent_avg": recent_avg,
            "baseline_avg": baseline_avg,
            "drift": drift,
            "significant": significance,
            "status": "drift_detected" if significance else "normal"
        }

    def _check_significance(self, drift: float) -> bool:
        """Check if drift is statistically significant."""
        # Simple threshold check
        return abs(drift) > 0.05  # 5% threshold

7. Evaluation Report Template

# LLM Evaluation Report

## Executive Summary
- **Overall Score**: 0.87/1.0
- **Status**: PASS (threshold: 0.80)
- **Trend**: +0.03 vs last month

## Performance by Task
| Task | Score | Samples | Status |
|------|-------|---------|--------|
| Customer Support | 0.91 | 150 | ✓ |
| Code Generation | 0.85 | 100 | ✓ |
| Data Analysis | 0.82 | 80 | ✓ |

## Failures Analysis
- **Common failure mode**: Factual inaccuracies in domain-specific queries
- **Affected tasks**: Data Analysis (12% failure rate)
- **Recommendation**: Add domain-specific fine-tuning

## Recommendations
1. Increase test coverage for Data Analysis
2. Implement factual verification layer
3. Schedule re-evaluation in 2 weeks

8. Production Checklist

• MTEB baseline established
• Task-specific test sets created (100+ samples each)
• A/B testing framework implemented
• LLM-as-judge pipeline operational
• Quality monitoring dashboard active
• Drift detection configured
• Evaluation reports automated
• Human review process defined
• Regression testing scheduled
• Version control for test sets

---

FAQ

Q: What benchmarks should I use for LLM evaluation?

A: Use MTEB for general capabilities, create task-specific tests for your domain, and run A/B tests comparing model versions.

Q: How to evaluate LLM output quality automatically?

A: Combine automated metrics (BLEU, ROUGE, BERTScore) with LLM-as-judge and periodic human review.

Q: What's the minimum test set size for reliable evaluation?

A: At least 100 representative samples per task. For production, maintain 500+ samples with regular refresh.

Q: How often should I re-evaluate my LLM?

A: Evaluate after every model update, monthly for production systems, and whenever output quality complaints increase.

Q: How to handle evaluation failures?

A: Log failures with context, categorize by type, prioritize fixes, and re-test after corrections.