title: LLM Application Monitoring — Observability for Production AI description: Complete monitoring stack for LLM apps: Prometheus metrics, quality tracking, cost analysis, Grafana dashboards, and alerting. Original ops guide. tags: [ai, llm, monitoring, observability] lang: en date: 2026-10-23 original: true faq: - q: "What metrics should I track for LLM applications?" a: "Track latency (TTFT, TPOT, total), token throughput, error rates, quality scores, cost per request, and cache hit rates." - q: "How to detect LLM hallucinations in production?" a: "Implement factual consistency checks, confidence scoring, and human-in-the-loop validation for critical outputs." - q: "What's the best way to track LLM costs?" a: "Log every API call with model, input/output tokens, and calculate costs using vendor pricing. Aggregate by user, feature, and time period." - q: "How to set up alerts for LLM degradation?" a: "Monitor quality metrics (not just latency), set thresholds for error rates, and track user feedback patterns."
LLM Application Monitoring — Observability for Production AI
Use Case: Build comprehensive monitoring for LLM applications: performance, quality, cost, and reliability.
1. Observability Framework
Three Pillars of LLM Observability
| Pillar | What to Track | Why It Matters |
|---|---|---|
| Performance | Latency, throughput, token rates | User experience, SLA compliance |
| Quality | Accuracy, relevance, coherence | Business value, user satisfaction |
| Cost | Token usage, API calls, budget | Financial control, ROI |
2. Key Metrics to Track
Performance Metrics
# Core latency metrics
from prometheus_client import Histogram, Counter, Gauge
# Request latency (total)
REQUEST_LATENCY = Histogram(
'llm_request_duration_seconds',
'LLM request latency',
['model', 'endpoint']
)
# Time to first token (TTFT) - critical for UX
TTFT = Histogram(
'llm_ttft_seconds',
'Time to first token',
['model']
)
# Tokens per second (throughput)
TOKEN_RATE = Histogram(
'llm_tokens_per_second',
'Token generation rate',
['model']
)
# Error rates
ERRORS = Counter(
'llm_errors_total',
'LLM errors',
['model', 'error_type']
)
Quality Metrics
# Quality scoring (0-1)
QUALITY_SCORE = Histogram(
'llm_quality_score',
'Output quality score',
['model', 'task_type']
)
# Hallucination rate
HALLUCINATION_RATE = Counter(
'llm_hallucinations_total',
'Detected hallucinations'
)
# User satisfaction (thumbs up/down)
USER_SATISFACTION = Counter(
'llm_user_satisfaction',
'User feedback',
['rating']
)
Cost Metrics
# Token usage
INPUT_TOKENS = Counter(
'llm_input_tokens_total',
'Input tokens consumed',
['model']
)
OUTPUT_TOKENS = Counter(
'llm_output_tokens_total',
'Output tokens generated',
['model']
)
# Cost calculation
COST_PER_REQUEST = Histogram(
'llm_cost_per_request_usd',
'Cost per request in USD'
)
3. Monitoring Architecture
Stack Overview
┌─────────────────────────────────────────────────────┐
│ Application Layer │
│ (API Gateway, LLM Client, Caching) │
└─────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Metrics Exporter │
│ (Prometheus Client, Custom Exporter) │
└─────────────────────────────────────────────────────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│Prometheus│ │Loki │ │Tempo │
│(Metrics) │ │(Logs) │ │(Traces) │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────┐
│ Grafana │
│ (Dashboards, Alerts, UI) │
└─────────────────────────────────┘
Implementation
from prometheus_client import start_http_server, CollectorRegistry
from opentelemetry import trace, metrics
from opentelemetry.exporter.prometheus import PrometheusMetricsExporter
import time
class LLMObservability:
def __init__(self):
self.start_time = time.time()
self.metrics = {
'requests_total': 0,
'tokens_input': 0,
'tokens_output': 0,
'errors_total': 0,
'cost_total': 0.0
}
# Start Prometheus HTTP server
start_http_server(8000)
async def track_request(self, model: str, prompt: str,
response: str, duration: float,
cost: float, error: bool = False):
"""Track a single LLM request."""
input_tokens = len(prompt.split()) * 1.3 # Approximate
output_tokens = len(response.split()) * 1.3
# Update metrics
self.metrics['requests_total'] += 1
self.metrics['tokens_input'] += input_tokens
self.metrics['tokens_output'] += output_tokens
self.metrics['cost_total'] += cost
if error:
self.metrics['errors_total'] += 1
# Record metrics
REQUEST_LATENCY.labels(model=model).observe(duration)
INPUT_TOKENS.labels(model=model).inc(input_tokens)
OUTPUT_TOKENS.labels(model=model).inc(output_tokens)
COST_PER_REQUEST.observe(cost)
def get_health(self) -> dict:
"""Health check endpoint."""
uptime = time.time() - self.start_time
error_rate = (self.metrics['errors_total'] /
max(self.metrics['requests_total'], 1))
return {
'status': 'healthy' if error_rate < 0.05 else 'degraded',
'uptime_seconds': uptime,
'error_rate': error_rate,
'requests_total': self.metrics['requests_total'],
'tokens_total': (self.metrics['tokens_input'] +
self.metrics['tokens_output'])
}
4. Grafana Dashboards
Dashboard 1: LLM Performance
{
"dashboard": {
"title": "LLM Performance Overview",
"panels": [
{
"title": "Request Latency (p95)",
"targets": [{
"expr": "histogram_quantile(0.95,
rate(llm_request_duration_seconds_bucket[5m]))"
}]
},
{
"title": "Tokens Per Second",
"targets": [{
"expr": "rate(llm_tokens_per_second_sum[5m]) /
rate(llm_tokens_per_second_count[5m])"
}]
},
{
"title": "Error Rate",
"targets": [{
"expr": "rate(llm_errors_total[5m]) /
rate(llm_requests_total[5m])"
}]
}
]
}
}
Dashboard 2: Cost Tracking
{
"dashboard": {
"title": "LLM Cost Analysis",
"panels": [
{
"title": "Daily Cost by Model",
"targets": [{
"expr": "sum by (model) (
rate(llm_cost_total[24h])
)"
}]
},
{
"title": "Cost Per Request",
"targets": [{
"expr": "llm_cost_per_request_usd_avg"
}]
},
{
"title": "Token Usage Trend",
"targets": [{
"expr": "rate(llm_tokens_total[24h])"
}]
}
]
}
}
5. Alerting Rules
# alerting_rules.yml
groups:
- name: llm_alerts
rules:
- alert: HighLLMLatency
expr: histogram_quantile(0.95,
rate(llm_request_duration_seconds_bucket[5m])) > 5
for: 10m
labels:
severity: warning
annotations:
summary: "LLM p95 latency above 5s"
- alert: HighErrorRate
expr: rate(llm_errors_total[5m]) /
rate(llm_requests_total[5m]) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "LLM error rate above 5%"
- alert: CostSpike
expr: rate(llm_cost_total[1h]) > 10
for: 15m
labels:
severity: warning
annotations:
summary: "Unusual cost increase detected"
- alert: QualityDegradation
expr: avg(llm_quality_score[1h]) < 0.7
for: 30m
labels:
severity: warning
annotations:
summary: "LLM output quality dropping"
6. Quality Monitoring
Automated Quality Checks
import openai
from sklearn.metrics import accuracy_score
class QualityMonitor:
def __init__(self, evaluator_model: str = "gpt-4"):
self.client = openai.OpenAI()
self.evaluator = evaluator_model
async def check_factual_consistency(
self,
question: str,
answer: str,
context: str
) -> dict:
"""Check if answer is consistent with context."""
prompt = f"""
Context: {context}
Question: {question}
Answer: {answer}
Is the answer fully supported by the context?
Return JSON: {{"consistent": true/false, "confidence": 0-1}}
"""
response = await self._call_evaluator(prompt)
return response
async def check_coherence(self, text: str) -> float:
"""Score text coherence (0-1)."""
# Use a fine-tuned model or heuristic
# Simple heuristic: sentence connectivity
sentences = text.split('. ')
if len(sentences) < 2:
return 0.5
# Check for logical flow
coherence_score = self._calculate_coherence(sentences)
return min(coherence_score, 1.0)
def _calculate_coherence(self, sentences: list) -> float:
"""Calculate coherence based on transitions."""
# Implementation depends on requirements
pass
Human-in-the-Loop Validation
import redis
class QualityWorkflow:
def __init__(self):
self.redis = redis.Redis()
self.pending_reviews = []
async def should_review(self, output: str, confidence: float) -> bool:
"""Decide if human review is needed."""
# Low confidence outputs need review
if confidence < 0.7:
return True
# Sensitive topics need review
if self._is_sensitive_topic(output):
return True
# Random sample for quality tracking
if hash(output) % 100 < 5: # 5% random sample
return True
return False
async def queue_for_review(self, item: dict):
"""Queue output for human review."""
review_id = f"review_{int(time.time())}"
self.redis.hset(f"review:{review_id}", mapping=item)
self.redis.lpush("review_queue", review_id)
return review_id
7. Cost Optimization Monitoring
Cost Per Feature
class CostTracker:
def __init__(self):
self.feature_costs = {}
def track(self, feature: str, cost: float, tokens: int):
"""Track cost by feature."""
if feature not in self.feature_costs:
self.feature_costs[feature] = {
'cost': 0.0,
'tokens': 0,
'requests': 0
}
self.feature_costs[feature]['cost'] += cost
self.feature_costs[feature]['tokens'] += tokens
self.feature_costs[feature]['requests'] += 1
def get_daily_report(self) -> dict:
"""Generate daily cost report."""
report = {}
for feature, data in self.feature_costs.items():
report[feature] = {
'total_cost': data['cost'],
'avg_cost_per_request': data['cost'] / max(data['requests'], 1),
'token_efficiency': data['tokens'] / max(data['requests'], 1)
}
return report
8. Production Checklist
- Prometheus metrics exporter running
- Grafana dashboards configured
- Alert rules defined and tested
- Quality monitoring implemented
- Cost tracking enabled
- Log aggregation set up (Loki/ELK)
- Distributed tracing configured (Tempo/Jaeger)
- Runbook created for common issues
- On-call rotation defined
- Weekly cost review scheduled
FAQ
Q: What metrics should I track for LLM applications?
A: Track latency (TTFT, TPOT, total), token throughput, error rates, quality scores, cost per request, and cache hit rates.
Q: How to detect LLM hallucinations in production?
A: Implement factual consistency checks, confidence scoring, and human-in-the-loop validation for critical outputs.
Q: What's the best way to track LLM costs?
A: Log every API call with model, input/output tokens, and calculate costs using vendor pricing. Aggregate by user, feature, and time period.
Q: How to set up alerts for LLM degradation?
A: Monitor quality metrics (not just latency), set thresholds for error rates, and track user feedback patterns.
Q: How often should I review LLM quality?
A: Implement automated checks for every request, human review for low-confidence outputs, and weekly trend analysis.