LLM Prompt Caching: Cutting Token Spend With the OpenRouter and vLLM Prefix Cache

Q: What metric should I watch for caching?

The provider's cache-hit/cache-read token ratio (e.g. cached_tokens in the OpenAI usage object or the vLLM APC metrics). A low hit rate means your prefix is unstable or your TTL is too short.

I run the inference layer for a document-heavy product: long PDFs, regulatory filings, and chat sessions that stretch across dozens of turns. For months I treated token cost as a fixed tax. Then I turned on prompt caching and watched the same traffic cost a fraction of what it did before. This article is the playbook I wish I had on day one — the mechanisms, the exact configs, the failure modes, and the bill that finally made me pay attention.

1. What prompt caching actually is, and the math that makes it worth it

Every LLM call carries a prefix — the system prompt, the tool definitions, the conversation history. If I send 10,000 tokens of the same prefix on every request, I pay for those 10,000 tokens every single time. Prompt caching changes that: the provider keeps a representation of that prefix in memory, and when I send the same bytes again, it charges me for the cached tokens at a much lower rate.

The arithmetic is where the magic hides. Imagine a 10K-token system prompt plus 500 tokens of user input:

No cache: (10,000 + 500) × full input rate every request.
With cache: 10,000 × cached rate + 500 × full input rate on a hit.

Cached input runs roughly 1/10 the cost of fresh input on OpenAI and Anthropic gives up to 90% off. The point is not that caching saves money — it is that the larger and more stable your prefix, the more the multiplier works for you. A 100-token system prompt saves almost nothing. A 10K-token context saves a lot.

2. Three paths: OpenAI, Anthropic, and vLLM APC

I have run all three, and they are not interchangeable — the mechanism differences decide which one you can even use.

OpenAI — automatic, prefix-based. OpenAI caches automatically on a stable prefix. I do not set a flag; I just keep the prefix stable and the provider caches it. The catch is that I cannot choose where the cache boundary is. It caches the longest matching prefix automatically, and the cached region expires on a rolling TTL (roughly 5–10 minutes of inactivity in practice).

Anthropic — explicit cache_control breakpoints. Anthropic does nothing unless I mark a block. I add "cache_control": {"type": "ephemeral"} to a block I want cached, and it caches everything up to that breakpoint. I control the boundary, the TTL (5 minutes default, 1 hour extended), and the layout. This is the most predictable of the three.

vLLM — automatic prefix caching (APC), server-side. Self-hosted vLLM keeps a prefix cache on the KV cache with --enable-prefix-caching. No prompt changes needed. The tradeoff is that the cache lives on each replica, so a request that lands on a different replica misses unless I add prefix-aware routing.

Path	Who decides the boundary	Cost signal	Extra work
OpenAI	Provider, automatic	`cached_tokens` in usage	Keep prefix stable
Anthropic	Me, via `cache_control`	`cache_read_input_tokens`	Place breakpoints, pick TTL
vLLM APC	Server, by prefix	`prefix_cache_hit_rate` gauge	Routing + replica sizing

3. OpenAI in practice: a stable system-prompt template

My first attempt at OpenAI caching failed silently. The reason was embarrassing: I was injecting the current date and a per-user session ID into the system prompt on every call. That made every prefix unique, so the cache never matched.

The fix was to split the prompt into two parts: a byte-identical shared prefix and a variable tail.

SYSTEM_PREFIX = """You are DocQA, a regulatory filings assistant.
Rules:
- Answer only from the provided context. If absent, say so.
- Cite section numbers inline like [§12.3].
- Never invent figures.
- If the filing is older than 2019, note the disclaimer first."""

SYSTEM_VARIABLE = f"\n\nCurrent filing: {filing_id}. Query date: {date.today().isoformat()}."

messages = [{"role": "system", "content": SYSTEM_PREFIX + SYSTEM_VARIABLE},
            {"role": "user", "content": user_query}]

The four rules I now treat as gospel:

The cached prefix must be byte-identical. Any interpolation — a date, a UUID, a nonce — breaks the match.
Make the prefix long enough. Short prefixes are cached, but the savings are noise. My prefixes are 6K–12K tokens.
Push all variable content after the stable block. Do not bury a timestamp in the middle of the system prompt.
Never mutate the template at runtime. Generate it once at process start and reuse the string.

usage = response.usage
print(usage.prompt_tokens, usage.cached_tokens)  # e.g. (10500, 10000)

When cached_tokens sits at ~95% of prompt_tokens, the cache is working. When it is zero, my prefix is unstable again.

4. Anthropic `cache_control` in practice: layout, TTL, and the mistakes

Anthropic requires me to be deliberate. I structure the request as multiple blocks and mark the ones I want cached. For a multi-turn chat, I cache the system prompt and the long context, then keep the latest user turn outside the cache so the last few tokens stay cheap to update:

{
  "model": "claude-sonnet-4-5",
  "max_tokens": 1024,
  "system": [
    {"type": "text", "text": "You are a triage assistant for incident tickets."},
    {"type": "text", "text": "Runbook: [long runbook text, ~8K tokens]",
     "cache_control": {"type": "ephemeral", "ttl": "1h"}}
  ],
  "messages": [
    {"role": "user", "content": "The payment webhook is retrying on node-7..."}
  ]
}

TTL selection. I use the default 5-minute TTL for interactive traffic where consecutive calls come within seconds. I use the 1-hour TTL for batch or nightly jobs where a request might arrive 20 minutes after the previous one. Choosing the wrong TTL is the most common silent failure — a 5-minute TTL with a 15-minute job cadence means I pay full price and never notice.

The four mistakes I made and fixed:

Marking the wrong block. I once put cache_control on the last message instead of the system block, so the cache captured the whole conversation including the variable tail — guaranteed miss on the next turn.
A cache block that changes every call. If the block contains a timestamp, it rewrites every time. Move the timestamp into the final non-cached block.
TTL too short for the job. Batch processing needs the long TTL.
Using cache_control on the final block. A breakpoint at the very end caches the entire prompt, so nothing is cheap on the next call. Place it before the last variable block.

The bill reflects the fix immediately: cache_read_input_tokens in the response usage jumps from near-zero to 80–90% of total input.

5. vLLM APC self-hosted: deploy command, routing, and metrics

For the self-hosted fleet I switched to vLLM, where prefix caching is a launch flag. The deployment command that works for me:

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --num-scheduler-steps 10

--enable-prefix-caching is the whole game — it reuses KV-cache blocks across requests with a shared prefix. The subtlety is that the cache is per-replica. With three replicas, a request that lands on replica B does not benefit from a prefix cached on replica A. I fixed this with a simple prefix-aware load balancer that hashes a stable part of the prompt (the system prompt + a coarse document ID) to pin repeat traffic to the same replica. The hit-rate metric I scrape from the vLLM Prometheus endpoint is vllm:prefix_cache_hit_rate — in production I see 0.85–0.95 on the document-heavy route and under 0.30 on the free-form chat route, which is expected, because that route has no stable prefix to speak of.

6. Monitoring hit rate: the metric that matters, and the counter-intuitive part

The metric I trust is the cache-read token ratio: provider-reported cached tokens divided by total input tokens. I alert on it:

- alert: PromptCacheHitRateLow
  expr: (cached_tokens / (cached_tokens + fresh_tokens)) < 0.7
  for: 30m
  annotations:
    summary: "Cache hit rate below 70% — check prefix stability or TTL"

The counter-intuitive point: the savings do not show up in the hit-rate itself; they show up in the token cost line. A route can have a 95% hit rate but trivial savings if the prefix is only 200 tokens. A route with a 60% hit rate on a 20K-token prefix can dominate the entire bill. I was initially optimizing the wrong number — I chased high hit rates on routes that cost almost nothing, while ignoring a chunky route where the hit rate was mediocre but the absolute savings were enormous. Watch cached_tokens × (fresh_rate − cached_rate) per route, not just the ratio.

7. Three real cases from production

Case 1 — RAG long-document Q&A, ~10K-token context. I send a fixed system prompt plus a retrieved-context block. On a 10K prefix, the per-request input cost dropped by ~90% once the prefix was stable. The gotcha: retrieved context changes with the query, so I cache only the system prompt and the document header, not the retrieved chunks. Hit rate sits at ~92%.

Case 2 — multi-turn chat with history truncation. I truncate history to the last N turns but keep the prefix stable by never injecting a per-turn timestamp into the system prompt. I place a cache_control breakpoint after the system prompt and one after the truncated history, leaving only the latest user turn uncached.

Case 3 — batch task processing (batch API + caching). I process thousands of similar records with the Batch API. Each request shares the same system prompt, so I use the 1-hour TTL on Anthropic and the batch discount on top. The combo was the single largest line-item reduction — the stable prefix was already cheap, and batch pricing cut the remainder in half.

8. The bill story: from $18K to $2.6K a month

The number that got my attention: $18,400 in one month, then $2,600 after three changes.

I stabilized the prefix (removed per-request interpolation from the system prompt). This alone captured most of the win — ~55% of the reduction.
I moved batch work to the Batch API with a 1-hour cache TTL. This took the remainder down by ~30% of the original.
I pinned vLLM replicas with prefix-aware routing on the self-hosted side, which cut the remaining self-hosted GPU spend by roughly another 15%.

What I learned: the wins were sequential, not additive to the same dollar. And the single biggest mistake was assuming the cache was working because the hit-rate panel looked healthy. It was healthy on a route that cost $40 a month, while the $18K route had a hit rate of 0.31 and I never looked.

9. Production checklist: six checkpoints before enabling caching

Isolate the prefix. Confirm the shared prefix is byte-identical across all traffic. Grep your codebase for any interpolation inside the system prompt.
Measure the prefix size. If it is under 1K tokens, caching is not the lever you think it is.
Place the boundary correctly. For Anthropic, verify the cache_control block is not the final block and does not contain variable content.
Pick the right TTL. Match it to your inter-request cadence, not your intuition.
Wire the metric. Capture cached_tokens / cache_read_input_tokens / vllm:prefix_cache_hit_rate before you roll out.
Alert on absolute savings, not just ratio. Compute cached_tokens × (fresh_rate − cached_rate) per route. That is the number the CFO cares about.

10. FAQ

Q: How much can prompt caching actually save? A: If your traffic repeats a large, stable system prompt or context, cached tokens cost a fraction of fresh ones. In my case it cut the monthly bill from $18K to $2.6K, but the hit rate must be measured — savings do not appear automatically.

Q: OpenAI, Anthropic, or vLLM caching — which one should I use? A: OpenAI caches automatically on a stable prefix; Anthropic requires explicit cache_control breakpoints; vLLM APC caches by prefix server-side. Pick the one matching your provider and architecture, not the newest feature.

Q: Why do I keep getting cache misses? A: Most misses come from an unstable prefix — dates, user IDs, or timestamps injected into the system prompt break the match. Keep the shared prefix byte-identical and put variable content after the cached block.

Q: What metric should I watch for caching? A: The provider's cache-hit/cache-read token ratio (e.g. cached_tokens in the OpenAI usage object or the vLLM APC metrics). A low hit rate means your prefix is unstable or your TTL is too short.

Q: Does prompt caching hurt output quality? A: No. Caching reuses the exact KV representation of a byte-identical prefix, so the output is deterministic relative to that prefix. The only way to change output is to change the prefix — which also means losing the cache.

Q: Can I cache across requests from different users? A: Yes, and that is where the biggest wins are. A shared system prompt is identical across all users, so the cache serves everyone. Only user-specific content must live after the cached block.