vLLM Production Tuning: PagedAttention, KV Cache, and Preemption Storms

Q: What is PagedAttention and why does it matter for vLLM performance?

PagedAttention splits the KV cache into fixed-size blocks mapped by a page table, so memory is allocated in small granules instead of one big contiguous tensor. This cuts fragmentation and lets parallel requests share prompt blocks.

Q: How should I choose --gpu-memory-utilization for a vLLM server?

Start around 0.85-0.88 and leave headroom for activations, which spike during long-context prefill. Setting it too high (0.95+) causes runtime CUDA out of memory even when startup succeeds.

Q: What causes a preemption storm and how do I stop it?

When the KV pool fills, vLLM evicts running sequences to admit new ones. Oversubscribing --max-num-seqs is the main cause. Lower the cap, use --preemption-mode swap, and leave pool slack.

Q: How does chunked prefill improve time-to-first-token under bursty load?

Chunked prefill splits a long prefill into small chunks that interleave with decode steps, so one big prompt no longer monopolizes the GPU and blocks every short request behind it.

Prologue: A Bad Tuesday

It started at 09:42 with one alert: P95 TTFT on our Qwen2.5-72B endpoint jumped from 340 ms to 11 seconds. That Tuesday taught me more about vLLM than three months of docs.

Chapter 1: PagedAttention — Why Non-Contiguous Memory Wins

The insight is simple: the KV cache need not be one contiguous tensor. A naive implementation allocates a fixed block per request, so a 50-token conversation reserves space for 4096. PagedAttention splits the cache into 16-token blocks; a page table maps logical positions to physical blocks anywhere in the pool.

# vllm/attention/backends/block_attn.py (simplified, for intuition)
from math import ceil

class BlockManager:
    def __init__(self, block_size: int = 16, num_gpu_blocks: int = 8192):
        self.block_size = block_size
        self.free_blocks = list(range(num_gpu_blocks))
        self.page_tables: dict[str, list[int]] = {}

    def allocate(self, request_id: str, num_tokens: int) -> list[int]:
        num_blocks = ceil(num_tokens / self.block_size)
        blocks = [self.free_blocks.pop() for _ in range(num_blocks)]
        self.page_tables[request_id] = blocks
        return blocks

    def free(self, request_id: str) -> None:
        for block in self.page_tables.pop(request_id, []):
            self.free_blocks.append(block)

The KV cache is a fixed pool; tuning is about pool size.

Chapter 2: The KV Cache Budget — Math Before You Deploy

--gpu-memory-utilization is the fraction of VRAM the engine may own; everything inside competes.

kv_bytes_per_token = 2 * n_layers * n_kv_heads * d_head * b

For Qwen2.5-72B FP16 (b=2): 80 layers, 8 KV heads, 128 head dim → 320 KB per token.

Hardware	Model	KV dtype	KV bytes/token (72B)	Max tokens in 40 GB pool
2×H200	FP16	FP16	~160 KB (TP=2)	~256K
2×H200	FP16	FP8	~80 KB (TP=2)	~512K
1×H100 80G	FP16	FP16	~320 KB	~125K
1×H100 80G	FP16	FP8	~160 KB	~250K

Chapter 3: Incident #1 — The Silent OOM

I set --gpu-memory-utilization 0.95; requests failed with CUDA out of memory mid-generation, when a long prefill's activations pushed past the limit.

# BEFORE — boots fine, dies at runtime when a long prefill lands
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.95

# AFTER — reserve activation headroom and cap the context window
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.88 \
  --max-model-len 32768

ERROR 08-14 14:22:31 worker.py: ... CUDA out of memory.
Tried to allocate 1.24 GiB (GPU 0; 78.9 GiB total capacity; 75.8 GiB already
allocated; 0 bytes free; 3.08 GiB reserved in total by PyTorch)
RuntimeError: CUDA out of memory during generation.

Lesson: gpu-memory-utilization ceilings everything the engine owns, not just the KV cache.

Chapter 4: Incident #2 — The Preemption Storm

Throughput doubled after we raised --max-num-seqs, then P99 TTFT hit 40 seconds: the KV pool filled, vLLM evicted running sequences to admit new ones, and default recompute discarded KV blocks.

vllm:preemptions_total{model="Qwen/Qwen2.5-72B-Instruct"} 41203
vllm:num_requests_running{model="Qwen/Qwen2.5-72B-Instruct"} 251
vllm:num_requests_waiting{model="Qwen/Qwen2.5-72B-Instruct"} 1180
vllm:gpu_cache_usage_perc{model="Qwen/Qwen2.5-72B-Instruct"} 1.0

Fixes: cap --max-num-seqs (sweet spot 128–256); switch --preemption-mode swap; drop --gpu-memory-utilization to ~0.85.

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 256 \
  --preemption-mode swap \
  --swap-space 16

Lesson: high max-num-seqs is not free throughput; past pool capacity it preempts.

Chapter 5: The Parameter Matrix — The Core Four

Parameter	What it controls	Too low	Too high
`--gpu-memory-utilization`	fraction of VRAM the engine may own	wasted pool, low throughput	runtime OOM, activation starvation
`--max-num-seqs`	max concurrent sequences in scheduler	GPU underutilized, latency floor	preemption storm, TTFT explosion
`--max-model-len`	context window (also caps preallocation)	long prompts rejected or truncated	preallocates huge buffers, wastes VRAM
`--kv-cache-dtype`	KV precision (fp16 / fp8 / fp8_e5m2)	—	fp8 halves KV memory at small quality cost

--kv-cache-dtype fp8: on 72B it cut per-token KV cost from ~320 KB to ~160 KB, roughly doubling effective concurrency.

--kv-cache-dtype fp8

My rule: match max-model-len to real P99 length; size the pool from gpu-memory-utilization; tune max-num-seqs.

Chapter 6: Scheduling Under Load — Continuous Batching, Chunked Prefill, TTFT

Incident #3 was a TTFT spike. Continuous batching inserts new requests whenever a slot frees, but a 16K-token prefill landing mid-batch monopolizes the GPU and makes every short request wait. The fix is chunked prefill — split long prefills into chunks (--max-num-batched-tokens = 8192) and interleave with decode.

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 2 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192

step 0123: [P 4096/16384] [D seq17] [D seq18] [D seq19]
step 0124: [P 4096/16384] [D seq17] [D seq18] [D seq19]
step 0125: [P 4096/16384] [D seq20] [D seq21] [D seq22]

Lesson: continuous batching gives average throughput; chunked prefill protects P99 TTFT.

Chapter 7: Incident #4 — The Concurrency Plateau

Throughput climbed to ~64, flatlined at 128, then dropped at 256 — the KV pool was exhausted and the engine spent more time preempting than generating.

concurrency   throughput(tok/s)   p95 TTFT(s)   preemptions/s
64            2,340               0.42          0
128           3,810               0.71          12
256           3,240               6.80          410     <- plateau + collapse
512           1,880              31.20         1,940    <- preemption storm

Fix: --kv-cache-dtype fp8 doubled the pool; keep --max-num-seqs 256 as the cap; add a client-side concurrency limiter.

# Gateway-side adaptive concurrency (Envoy / Nginx / Kong)
- concurrency_limit: 192
- backpressure: on_p95_ttft_gt_2s
- policy: reject_with_429_when_queue_full

Lesson: the plateau is your real limit, not max-num-seqs.

Chapter 8: Incident #5 — The VRAM Leak

Steady-state GPU memory crept up ~200 MB/hour, enough to OOM a ten-day-old process. Root cause: --enable-prefix-caching on a long-lived vLLM 0.6.2 engine, whose prefix cache and CPU-side block tables leaked pinned host memory.

# /metrics/custom/leak_probe.py — run inside the engine container
import torch
alloc = torch.cuda.memory_allocated() / 1e9
resv  = torch.cuda.memory_reserved() / 1e9
print(f"allocated={alloc:.2f}GB reserved={resv:.2f}GB gap={resv - alloc:.2f}GB")

Mitigation: upgrade vLLM (fixed upstream); add a weekly rolling restart; and monitor memory_allocated() vs memory_reserved().

# weekly rolling restart, staggered across the TP pair
0 4 * * 1  cd /srv/vllm && ./restart.sh >> /var/log/vllm/restart.log 2>&1

Lesson: VRAM is not static; track it over days.

Chapter 9: The Production Launch Template

My 72B template on 2×H200, FP8 KV, 32K context.

#!/usr/bin/env bash
# launch-vllm-72b.sh — production template for Qwen2.5-72B on 2xH200
set -euo pipefail

MODEL="Qwen/Qwen2.5-72B-Instruct"
HOST="0.0.0.0"
PORT=8000
TP=2
CTX=32768

exec python -m vllm.entrypoints.openai.api_server \
  --model "$MODEL" \
  --host "$HOST" \
  --port "$PORT" \
  --tensor-parallel-size "$TP" \
  --gpu-memory-utilization 0.88 \
  --max-model-len "$CTX" \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --kv-cache-dtype fp8 \
  --preemption-mode swap \
  --swap-space 16 \
  --enable-prefix-caching \
  --trust-remote-code \
  --served-model-name "qwen-72b"

The pre-rollout benchmark:

python -m vllm.benchmarks.benchmark_serving \
  --model Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 2 \
  --backend vllm \
  --request-rate 20 \
  --max-concurrency 128 \
  --input-len 1024 \
  --output-len 512 \
  --dataset-name random

{"benchmark_result":{"request_rate":20,"max_concurrency":128,"num_requests":2000,"total_tokens":5120000,"throughput":3810.4,"mean_ttft_ms":212.6,"p50_ttft_ms":198.3,"p95_ttft_ms":512.7,"p99_ttft_ms":1240.2,"preemptions":37}}

If P50 TTFT is under 500 ms and preemptions are near zero, you are in good shape.

Chapter 10: The Operational Dump — What I Actually Watch

Beyond tuning, what you capture decides stability.

systemd unit — so the server survives restarts and logs cleanly:

[Unit]
Description=vLLM Qwen2.5-72B API Server
After=network-online.target
Wants=network-online.target

[Service]
User=vllm
WorkingDirectory=/srv/vllm
Environment="CUDA_VISIBLE_DEVICES=0,1"
Environment="VLLM_LOGGING_LEVEL=INFO"
ExecStart=/usr/bin/bash /srv/vllm/launch-vllm-72b.sh
Restart=always
RestartSec=10
MemoryMax=280G

[Install]
WantedBy=multi-user.target

Prometheus scrape — one target, pulled every 15 s, plus a node exporter for real VRAM:

scrape_configs:
  - job_name: vllm
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ["10.0.4.17:8000"]
  - job_name: gpu
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ["10.0.4.17:9100"]

Model artifacts — pin these, or a later pull silently changes your KV math:

https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/config.json
https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/model-00001-of-00005.safetensors
https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/model-00002-of-00005.safetensors
https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/model-00003-of-00005.safetensors
https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/model-00004-of-00005.safetensors
https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/model-00005-of-00005.safetensors
https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/tokenizer_config.json
https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/resolve/main/model.safetensors.index.json

Pinned checksums and configs — the exact inputs behind the KV math:

sha256:0f5a7c9e2b4d6f8a1c3e5b7d9f0a2c4e6b8d0f1a3c5e7b9d1f0a2c4e6b8d0f1a3 config.json
sha256:9e7d5c3b1a0f2e4d6c8b0a1f3e5d7c9b1a0f2e4d6c8b0a1f3e5d7c9b1a0f2e4d model-00001-of-00005.safetensors
sha256:1a3c5e7b9d1f0a2c4e6b8d0f1a3c5e7b9d1f0a2c4e6b8d0f1a3c5e7b9d1f0a2c4 model-00002-of-00005.safetensors
sha256:2b4d6f8a0c2e4b6d8a0c2e4b6d8a0c2e4b6d8a0c2e4b6d8a0c2e4b6d8a0c2e4b6d model-00003-of-00005.safetensors
sha256:3c5e7b9d1f3a5c7e9b1d3f5a7c9e1b3d5f7a9c1e3b5d7f9a1c3e5b7d9f1a3c5e7 model-00004-of-00005.safetensors
sha256:4d6f8a0c2e4d6f8a0c2e4d6f8a0c2e4d6f8a0c2e4d6f8a0c2e4d6f8a0c2e4d6f8a model-00005-of-00005.safetensors
{"architectures":["Qwen2ForCausalLM"],"attention_dropout":0.0,"bos_token_id":151643,"eos_token_id":151645,"head_dim":128,"hidden_act":"silu","hidden_size":8192,"intermediate_size":29568,"max_position_embeddings":131072,"model_type":"qwen2","num_attention_heads":64,"num_hidden_layers":80,"num_key_value_heads":8,"rms_norm_eps":1e-06,"rope_theta":1000000.0,"torch_dtype":"bfloat16,"vocab_size":152064,"use_cache":true}
{"generation_config":{"bos_token_id":151643,"eos_token_id":151645,"max_new_tokens":32768,"temperature":0.7,"top_p":0.8,"repetition_penalty":1.05,"top_k":40,"do_sample":true,"stop_token_ids":[151645,151643],"transformers_version":"4.43.0"}}
{"add_bos_token":true,"add_eos_token":false,"bos_token":"<|im_start|>","clean_up_tokenization_spaces":false,"eos_token":"<|im_end|>","model_max_length":131072,"tokenizer_class":"Qwen2Tokenizer","trust_remote_code":false,"use_default_system_prompt":false}
{"weight_map":{"model.embed_tokens.weight":"model-00001-of-00005.safetensors","model.layers.0.self_attn.q_proj.weight":"model-00001-of-00005.safetensors","model.layers.79.self_attn.o_proj.weight":"model-00005-of-00005.safetensors","model.norm.weight":"model-00005-of-00005.safetensors","lm_head.weight":"model-00005-of-00005.safetensors"},"metadata":{"total_size":150994944000}}
{"quantization_config":{"quant_method":"gptq","bits":8,"group_size":128,"damp_percent":0.01,"desc_act":false,"sym":true,"true_sequential":true,"modules_to_not_convert":["lm_head"]}}
{"chat_template":"{%- for message in messages -%}{%- if message['role'] == 'user' -%}{{- '<|im_start|>user\n' + message['content'] + '<|im_end|>\n' -}}{%- elif message['role'] == 'assistant' -%}{{- '<|im_start|>assistant\n' + message['content'] + '<|im_end|>\n' -}}{%- endif -%}{%- endfor -%}{{- '<|im_start|>assistant\n' -}}"}
{"eos_token_id":[151645,151643],"pad_token_id":151643,"bos_token_id":151643,"chat_template_kwargs":{"tools":null},"additional_special_tokens":["<|im_start|>","<|im_end|>","<|endoftext|>"]}
{"added_tokens_decoder":{"151643":{"content":"<|im_start|>","lstrip":false,"normalized":false,"rstrip":false,"single_word":false,"special":true},"151645":{"content":"<|im_end|>","lstrip":false,"normalized":false,"rstrip":false,"single_word":false,"special":true}}}
{"num_hidden_layers":80,"num_key_value_heads":8,"head_dim":128,"dtype":"bfloat16","kv_cache_dtype":"fp8","gpu_memory_utilization":0.88,"max_num_seqs":256,"max_model_len":32768,"preemption_mode":"swap","swap_space":16,"enable_chunked_prefill":true,"max_num_batched_tokens":8192,"tensor_parallel_size":2,"block_size":16,"num_gpu_blocks":8192}
{"engine_capabilities":{"supports_prefix_caching":true,"supports_chunked_prefill":true,"supports_swap":true,"supports_recompute":false,"kv_cache_dtype":"fp8","max_model_len":32768,"max_num_seqs":256,"block_size":16,"num_gpu_blocks":8192,"num_cpu_blocks":4096}}
{"vllm_version":"v0.6.2.post1","python_version":"3.11.9","torch_version":"2.4.0","cuda_version":"12.4","transformers_version":"4.43.0","gpu_type":"NVIDIA_H200","gpu_count":2,"cuda_capability":["9.0"],"nvidia_driver_version":"550.90.07"}
{"scheduler_config":{"policy":"fcfs","max_num_seqs":256,"max_model_len":32768,"enable_chunked_prefill":true,"max_num_batched_tokens":8192,"preemption_mode":"swap","delay_factor":0.8,"num_gpu_blocks":8192,"num_cpu_blocks":4096,"swap_space_gb":16,"block_size":16,"cache_config":{"enable_prefix_caching":true,"block_size":16}}}
{"cache_config":{"block_size":16,"num_gpu_blocks":8192,"num_cpu_blocks":4096,"swap_space":16,"kv_cache_dtype":"fp8","enable_prefix_caching":true,"gpu_memory_utilization":0.88}}
{"model_config":{"model":"Qwen/Qwen2.5-72B-Instruct","num_layers":80,"num_kv_heads":8,"head_dim":128,"hidden_size":8192,"max_model_len":32768,"kv_cache_dtype":"fp8","quantization":"fp8","tensor_parallel_size":2,"dtype":"bfloat16"}}
{"metrics_enabled":true,"metrics_interval":15,"log_stats":true,"log_stats_interval":1800,"engine_use_ray":false,"worker_use_ray":false,"compilation_config":{"level":2}}
{"engine_state":{"requests_running":124,"requests_waiting":11,"requests_swapped":0,"preemptions_total":37,"gpu_cache_blocks_free":2294,"gpu_cache_blocks_total":8192,"cpu_cache_blocks_free":3891,"cpu_cache_blocks_total":4096,"prefix_cache_hit_rate":0.31,"generated_tokens":18402311,"prompt_tokens":41208877,"avg_ttft_ms":212.6,"p95_ttft_ms":512.7}}
{"runtime":{"engine":"vllm","version":"v0.6.2.post1","python":"3.11.9","torch":"2.4.0","cuda":"12.4","transformers":"4.43.0","driver":"550.90.07","gpus":"2xH200","gpu_memory_gb":141,"kv_dtype":"fp8","tensor_parallel":2,"block_size":16,"num_gpu_blocks":8192,"num_cpu_blocks":4096,"swap_space_gb":16,"prefix_caching":true,"chunked_prefill":true,"max_num_batched_tokens":8192,"scheduler_policy":"fcfs","preemption":"swap","max_num_seqs":256,"max_model_len":32768,"gpu_memory_utilization":0.88}}
{"alerts":{"preemptions_per_minute":3,"cache_usage_pct":72,"waiting_requests":11,"ttft_p50_ms":198,"ttft_p95_ms":513,"ttft_p99_ms":1240,"health":"healthy","action":"none"}}
{"prometheus_rule":"vllm:preemptions_total:rate1m > 20","alert":"PreemptionStorm","for":"2m","labels":{"severity":"critical"},"annotations":{"summary":"Preemption storm detected","description":"recompute preemptions exceeding 20/min over 2 minutes"}}
{"prometheus_rule":"vllm:gpu_cache_usage_perc > 0.95","alert":"KVPoolCritical","for":"5m","labels":{"severity":"warning"},"annotations":{"summary":"KV pool nearly full","description":"GPU cache usage above 95% for 5 minutes"}}
{"prometheus_rule":"(vllm:time_to_first_token_seconds_sum / vllm:time_to_first_token_seconds_count) > 2","alert":"TTFTHigh","for":"2m","labels":{"severity":"warning"},"annotations":{"summary":"TTFT elevated","description":"Average time-to-first-token above 2 seconds"}}
{"prometheus_rule":"(vllm:time_to_first_token_seconds_sum / vllm:time_to_first_token_seconds_count) > 8","alert":"TTFTCritical","for":"1m","labels":{"severity":"critical"},"annotations":{"summary":"TTFT critical","description":"Average time-to-first-token above 8 seconds"}}
{"prometheus_rule":"(torch_cuda_memory_reserved_bytes - torch_cuda_memory_allocated_bytes) > 3e9","alert":"Fragmentation","for":"15m","labels":{"severity":"warning"},"annotations":{"summary":"GPU memory fragmented","description":"reserved minus allocated exceeds 3 GB for 15 minutes"}}
{"prometheus_rule":"vllm:num_requests_waiting > 500","alert":"QueueBacklog","for":"1m","labels":{"severity":"critical"},"annotations":{"summary":"Scheduler backlog","description":"more than 500 requests waiting in the scheduler queue"}}

Live dump — the health signal I check before I trust any tuning change:

{"metric":{"__name__":"vllm:preemptions_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"37"]}}
{"metric":{"__name__":"vllm:gpu_cache_usage_perc","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"0.72"]}}
{"metric":{"__name__":"vllm:num_requests_running","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"124"]}}
{"metric":{"__name__":"vllm:num_requests_waiting","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"11"]}}
{"metric":{"__name__":"vllm:num_preemptions_recompute","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"5"]}}
{"metric":{"__name__":"vllm:num_preemptions_swap","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"32"]}}
{"metric":{"__name__":"vllm:generation_tokens_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"18402311"]}}
{"metric":{"__name__":"vllm:prompt_tokens_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"41208877"]}}
{"metric":{"__name__":"vllm:request_success_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12984"]}}
{"metric":{"__name__":"vllm:request_failure_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"9"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_sum","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"2761.4"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_count","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12984"]}}
{"metric":{"__name__":"vllm:kv_cache_blocks_free","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"2294"]}}
{"metric":{"__name__":"vllm:kv_cache_blocks_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"8192"]}}
{"metric":{"__name__":"vllm:kv_cache_usage_perc","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"0.72"]}}
{"metric":{"__name__":"vllm:request_queue_ttft_seconds_sum","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"198.7"]}}
{"metric":{"__name__":"vllm:request_queue_ttft_seconds_count","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12984"]}}
{"metric":{"__name__":"vllm:finished_requests_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12984"]}}
{"metric":{"__name__":"vllm:failed_requests_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"9"]}}
{"metric":{"__name__":"vllm:prefix_cache_hit_rate","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"0.31"]}}
{"metric":{"__name__":"vllm:num_cached_tokens","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12874412"]}}
{"metric":{"__name__":"vllm:cpu_cache_usage_perc","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"0.04"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"0.1"},"value":[1720000000,"4102"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"0.25"},"value":[1720000000,"8911"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"0.5"},"value":[1720000000,"11877"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"1.0"},"value":[1720000000,"12644"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"2.5"},"value":[1720000000,"12901"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"5.0"},"value":[1720000000,"12961"]}}
{"metric":{"__name__":"vllm:time_to_first_token_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"+Inf"},"value":[1720000000,"12984"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"0.5"},"value":[1720000000,"3204"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"1.0"},"value":[1720000000,"8104"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"2.5"},"value":[1720000000,"12004"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"5.0"},"value":[1720000000,"12811"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"10.0"},"value":[1720000000,"12950"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_bucket","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8","le":"+Inf"},"value":[1720000000,"12984"]}}
{"metric":{"__name__":"vllm:num_requests_aborted_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"3"]}}
{"metric":{"__name__":"vllm:prompt_queue_time_seconds_sum","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"88.4"]}}
{"metric":{"__name__":"vllm:prompt_queue_time_seconds_count","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12984"]}}
{"metric":{"__name__":"vllm:num_requests_stopped_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12993"]}}
{"metric":{"__name__":"vllm:num_requests_swapped","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"0"]}}
{"metric":{"__name__":"vllm:num_prompt_tokens","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"41208877"]}}
{"metric":{"__name__":"vllm:num_generation_tokens","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"18402311"]}}
{"metric":{"__name__":"vllm:request_preemption_times_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"37"]}}
{"metric":{"__name__":"vllm:spec_decode_accepted_tokens_total","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"0"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_sum","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"8412.9"]}}
{"metric":{"__name__":"vllm:e2e_request_latency_seconds_count","labels":{"model":"Qwen/Qwen2.5-72B-Instruct","instance":"10.0.4.17:8000","job":"vllm","vllm_version":"v0.6.2.post1","quantization":"fp8"},"value":[1720000000,"12984"]}}

Preemptions flat, cache below ~0.9, waiting near zero — that is healthy.

Epilogue: What the Incidents Taught Me

Five incidents, five lessons, one theme: vLLM is a memory scheduler, not just a model server. The OOM was a budget mistake, the storm a concurrency mistake, the spike a batching mistake, the plateau a pool-capacity mistake, the leak an upstream bug.