title: High-Concurrency LLM Inference with vLLM — Production Guide
description: vLLM deployment and tuning: PagedAttention原理, multi-GPU parallelism, continuous batching, TP/PP configuration, benchmarking. Original tutorial with 6 production pitfalls.
tags: [ai, vllm, llm, serving, gpu]
lang: en
date: 2026-10-23
original: true
faq:
- q: "How to choose between vLLM and TensorRT-LLM?"
a: "vLLM is easier to deploy, has an active community, and broad model support. TensorRT-LLM has higher performance ceiling but requires CUDA tuning. Choose TensorRT-LLM for latency-sensitive scenarios, vLLM for rapid deployment."
- q: "How to configure tensor parallelism (TP) across multiple GPUs?"
a: "Add --tensor-parallel-size <n> at startup, where n is the GPU count. Example for 4 GPUs: vllm serve meta-llama/Meta-Llama-3-70B --tensor-parallel-size 4."
- q: "What to do when vLLM runs out of GPU memory?"
a: "① Reduce max_model_len ② Enable --enable-chunked-prefill ③ Switch to a smaller quantized model ④ Increase GPU memory or reduce concurrency."
- q: "How to achieve high throughput with low latency?"
a: "① Increase batch_size ② Enable continuous batching ③ Use GPU FP8 quantization ④ Enable GPU prefetching. Target: P99 < 200ms, throughput > 1000 token/s."
- q: "How to handle cold start latency?"
a: "Use tensorizer load format (--load-format tensorizer), enable Ray distributed executor (--distributed-executor-backend ray), and pre-warm KV cache with dummy requests before serving traffic."
High-Concurrency LLM Inference with vLLM — Production Guide
Use Case: Deploy LLM inference services in production to support API calls, RAG retrieval, and Agent execution under high concurrency.
1. What is vLLM and Why Choose It
vLLM (Virtual Large Language Model) is an open-source inference engine developed by UC Berkeley. Its core innovation is PagedAttention — managing KV Cache as virtual memory pages, solving the severe GPU memory fragmentation problem in traditional LLM inference.
Comparison with Other Inference Frameworks:
| Framework | Pros | Cons | Use Case |
|---|---|---|---|
| vLLM | Easy to deploy, large community, broad model support | Slightly lower peak performance than TensorRT | General production deployment |
| TensorRT-LLM | Extreme performance, deep operator optimization | Steep learning curve, hardware-bound | Latency-sensitive scenarios |
| TGI | HuggingFace official, simple | Fewer features | HF ecosystem users |
| Ollama | Minimalist, local-first | No high concurrency support | Personal/edge scenarios |
2. Minimal Working Deployment (Single GPU)
2.1 Docker Deployment
# Pull the image
docker pull vllm/vllm-openai:latest
# Start (using llama-3.1-8b as example)
docker run --gpus all -p 8000:8000 \
-v /data/models:/root/.cache/huggingface \
vllm/vllm-openai:latest \
/bin/sh -c "vllm serve meta-llama/Meta-Llama-3.1-8B \
--host 0.0.0.0 \
--port 8000 \
--max-model-len 8192 \
--gpu-memory-utilization 0.9"
2.2 Verify the Service
# OpenAI-compatible endpoint
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 100
}'
3. Multi-GPU Parallel Configuration
3.1 Tensor Parallelism (TP)
Splits model weights across multiple GPUs. Best for single-request latency optimization.
# 4-card TP
vllm serve meta-llama/Meta-Llama-3-70B \
--tensor-parallel-size 4 \
--max-model-len 16384
# Verify TP is active
nvidia-smi # Should see all 4 GPUs with memory usage
TP Configuration Guidelines: - 7B model: 1-2 GPUs sufficient - 70B model: 4-8 GPUs - 405B model: 8+ GPUs + PP
3.2 Pipeline Parallelism (PP)
Splits model layers across different GPUs. Best when GPU memory is insufficient but GPU count is adequate.
3.3 TP + PP Hybrid
# 8 GPUs: TP=4, PP=2
vllm serve meta-llama/Meta-Llama-3-405B \
--tensor-parallel-size 4 \
--pipeline-parallel-size 2
4. Performance Tuning Parameters
4.1 Key Parameters Quick Reference
| Parameter | Default | Recommended | Description |
|---|---|---|---|
--max-model-len |
4096 | 8192-32768 | Max sequence length, affects memory |
--gpu-memory-utilization |
0.9 | 0.85-0.95 | GPU memory utilization rate |
--max-num-seqs |
256 | 128-512 | Max concurrent requests |
--chunked-prefill-size |
8192 | 4096-16384 | Prefill chunk size |
--disable-custom-all-reduce |
False | True (multi-GPU) | Disable custom NCCL |
4.2 Production-Ready Configuration
docker run --gpus all -p 8000:8000 \
-e VLLM_USE_PREFILL=False \
vllm/vllm-openai:latest \
/bin/sh -c "vllm serve meta-llama/Meta-Llama-3.1-70B \
--host 0.0.0.0 \
--port 8000 \
--max-model-len 32768 \
--max-num-seqs 256 \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--enable-chunked-prefill \
--chunked-prefill-size 8192 \
--disable-custom-all-reduce"
5. Performance Benchmarking
5.1 Benchmark Script
# benchmark_vllm.py
import asyncio
import aiohttp
import time
async def benchmark(base_url: str, model: str, num_requests: int = 100):
latencies = []
async with aiohttp.ClientSession() as session:
for i in range(num_requests):
start = time.perf_counter()
async with session.post(
f"{base_url}/v1/chat/completions",
json={
"model": model,
"messages": [{"role": "user", "content": "Write a 200-word essay about Kubernetes."}],
"max_tokens": 200,
"temperature": 0.7
}
) as resp:
await resp.json()
latencies.append(time.perf_counter() - start)
# Statistics
import numpy as np
p50 = np.percentile(latencies, 50)
p90 = np.percentile(latencies, 90)
p99 = np.percentile(latencies, 99)
throughput = num_requests / sum(latencies)
print(f"Requests: {num_requests}")
print(f"P50 Latency: {p50*1000:.1f}ms")
print(f"P90 Latency: {p90*1000:.1f}ms")
print(f"P99 Latency: {p99*1000:.1f}ms")
print(f"Throughput: {throughput:.2f} req/s")
if __name__ == "__main__":
asyncio.run(benchmark("http://localhost:8000", "meta-llama/Meta-Llama-3.1-70B"))
5.2 Metric Interpretation
| Metric | Healthy Threshold | Needs Optimization |
|---|---|---|
| P99 Latency | < 500ms | > 1s |
| Throughput | > 50 req/s | < 10 req/s |
| GPU Utilization | > 80% | < 50% |
| Memory Usage | 85-95% | < 70% or > 99% |
6. Common Troubleshooting (6 Pitfalls)
Issue 1: OOM — Out of GPU Memory
Symptom: RuntimeError: CUDA out of memory
Debug:
Fix:
- Reduce --max-model-len
- Enable --enable-chunked-prefill
- Lower --gpu-memory-utilization
Issue 2: TP Communication Bottleneck
Symptom: Multi-GPU inference is slower than single-GPU
Cause: GPUs communicating over PCIe with insufficient bandwidth
Fix:
- Use NVLink-connected GPUs (A100/H100)
- Or switch to PP instead of TP
- Adjust --tensor-parallel-size
Issue 3: Long Requests Queue Severely
Symptom: Short requests have normal latency, long requests queue up
Cause: Continuous batching not enabled or misconfigured
Fix:
Issue 4: Token Generation Speed Fluctuates
Symptom: Some requests are fast, others extremely slow
Cause: KV Cache memory fragmentation during prefill phase
Fix:
- Increase --gpu-memory-utilization to 0.95
- Fix batch size: --max-num-seqs 128
Issue 5: Quality Degradation After Quantization
Symptom: FP16 produces good output, INT8/FP8 output quality is poor
Fix:
- Use vLLM's native INT8 quantization (--quantization fp8)
- Avoid loading manually quantized models
- Or use AWQ quantization: --quantization awq
Issue 6: Slow Cold Start
Symptom: First request latency > 10s
Cause: Model loading + KV Cache warming
Fix:
7. Further Reading
FAQ
--tensor-parallel-size <n> at startup, where n is the GPU count. Example for 4 GPUs: vllm serve meta-llama/Meta-Llama-3-70B --tensor-parallel-size 4.