Prompt Caching Doesn't Cache Prompts

Prompt caching is the single biggest cost lever for anyone building agents on LLM APIs. Turn it on and your repeated input costs drop up to 90%, your time-to-first-token drops with them, and your 30-step agent workflow goes from $4.50 to $0.62.

Despite the name, it doesn't cache your prompt. It doesn't cache your tokens either. What it caches is something else entirely, and understanding that changes how you structure every API call you make.

What Actually Happens When You Call an LLM

When you send a prompt to an LLM API, the model goes through two distinct phases before you get a single word back.

Tokenization chops your text into integer IDs. "Hello world" becomes [9906, 1917]. Same input, same output, every time.

Prefill is the expensive setup. The model reads through every token in your prompt and builds an internal working state: a representation of what each token means in the context of every token before it. This is where the GPU does most of its work.

Decode is the cheap part. With the working state ready, the model generates output tokens one at a time, streaming them back to you.

That working state in the middle has a name: the KV cache. The K and V refer to internal vectors the model computes at every layer (a follow-up post will unpack the mechanics). For everything in this post, the intuition is enough: the KV cache is the expensive thing prefill produces, and it's what prompt caching stores and reloads.

When you see this in your API response:

"usage": {
  "input_tokens": 47,
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 98432,
  "output_tokens": 312
}

Those 98,432 cache_read_input_tokens mean the provider loaded a stored KV cache instead of rerunning prefill. You skipped the expensive part.

Why the Cache Is the Whole Game

A 50K-token prompt without caching means the model rebuilds its entire working state on every request. That's full prefill, every time, even if 49,999 of those tokens are identical to last call.

Prompt caching changes that. The first time you send a prompt with caching enabled, the provider stores the KV cache from prefill. On the next request, if your prompt starts with the same tokens in the same order, the provider loads the stored state instead of recomputing it.

For agents, where every round trip sends the same system prompt, same tools, same conversation history plus one new message, prompt caching means you only pay for prefill once. Every subsequent turn loads the stored state and jumps straight to generating a response.

The size of that state is also why caching is worth so much. For a large model like Llama 3 70B, each token of cached state is roughly 320 KB. A 4K-token system prompt = ~1.25 GB. A 128K context window = ~40 GB. Recomputing that on every call is what makes uncached agents painfully expensive. Loading it from storage is comparatively trivial.

Why Prefix Matching Must Be Exact

One rule governs everything else: the cache only matches from the start, and the first point of divergence kills everything after it.

That's because the working state at position t encodes the full context of every token before it, not token t alone. Change token 5, and every cached value from position 5 onward was computed against a prefix that no longer exists.

"Analyze this code"   → tokens: [2127, 56956, 420, 2082]
"Analyze this code "  → tokens: [2127, 56956, 420, 2082, 220]
                                                         ^^^
                                    Trailing space = extra token
                                    = different prefix = cache MISS

A trailing space. A reordered JSON key. A timestamp that ticked forward. Each one produces a different token sequence, and the cached computation no longer applies.

Which means prompt structure directly determines cache hit rate.

Everything left of the first mismatch is cached. Everything right gets recomputed. Push stable content to the front.

How Providers Implement This

Same mechanism: prefix-matched KV cache. Different API surfaces. The mechanism is stable, but the specific numbers below (prices, TTLs, model names, token minimums) move fast: these are current as of July 2026, so check the provider docs before you rely on a figure.

Anthropic

Anthropic offers two modes. Automatic caching (released February 2026, now available everywhere except Bedrock) needs a single cache_control field at the top of the request. The system advances the breakpoint forward as the conversation grows, checking the last 20 content blocks for a match:

{
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "cache_control": {"type": "ephemeral"},
  "system": "You are a helpful assistant...",
  "messages": [...]
}

For multi-tenant agents, explicit breakpoints (capped at 4) let you create concentric caching layers: tools cached across all requests, context cached per tenant, conversation cached per session.

{
  "model": "claude-sonnet-5",
  "cache_control": {"type": "ephemeral"},
  "tools": [
    {"name": "search", "...": "..."},
    {"name": "write_file", "...",
     "cache_control": {"type": "ephemeral"}}          // ← Breakpoint 1
  ],
  "system": [
    {"type": "text", "text": "You are a coding assistant...",
     "cache_control": {"type": "ephemeral"}},          // ← Breakpoint 2
    {"type": "text", "text": "<tenant's codebase context>",
     "cache_control": {"type": "ephemeral"}}           // ← Breakpoint 3
  ],
  "messages": [...]                                    // ← Breakpoint 4 (auto)
}

Breakpoints themselves are free, and each cache hit refreshes the timer at no cost (write and read pricing in the table below). The minimum cacheable prompt is per-model, and it keeps falling: 4,096 tokens on Opus 4.5 / 4.6 and Haiku 4.5, 2,048 on Opus 4.7, 1,024 on Opus 4.8 and the Sonnet line, down to 512 on Fable 5 and Mythos 5. Every generation lowers the bar.

OpenAI

OpenAI caches automatically once a prompt passes its minimum size, with no code changes and no write surcharge. On GPT-5.5 and newer the cache now lasts 24 hours by default, and the older 5-10 minute in_memory cache isn't supported on those. Pass an optional prompt_cache_key so requests with the same prefix land on the same machines, which is where the cache actually lives.

Google Gemini

Gemini splits the same idea in two. Implicit caching is on by default for 2.5 and newer models, with no setup required. Explicit caching uses named cache objects you create and manage, with custom TTLs and a storage charge that scales with size and duration. Note that it lives on the older generateContent API; Google's newer Interactions API is implicit-only:

cache = client.caches.create(
    model="gemini-3.5-flash",
    config=types.CreateCachedContentConfig(
        contents=[large_document],
        ttl="7200s",   # 2 hours
    ),
)

At a glance

What This Means for Agents

If you're building agents with tool-calling loops (20, 30, 50 round trips per task), cache behavior dominates your cost and latency. These patterns come from building exactly that.

Never change your tool set mid-conversation. Tool definitions live at the very front of the cached prefix. Add or remove one tool, and you invalidate everything downstream: system prompt, conversation history, all of it. Keep all tools in every request. Control behavior through system messages instead. Claude Code does this: EnterPlanMode and ExitPlanMode are tools, not tool-set swaps.

Put anything dynamic in messages, not the system prompt. Timestamps, mode flags, session IDs: if they live in the system prompt, every change invalidates the system cache and everything after it. Move dynamic state into a system message within the messages array. The system prompt stays frozen. The cache stays warm.

Use deterministic serialization. JSON key order isn't guaranteed. Different orderings produce different tokens, which means cache misses, even when the data is semantically identical.

# Cache-hostile
json.dumps(tool_result)

# Cache-friendly
json.dumps(tool_result, sort_keys=True)

Keep conversation history append-only. Edit or truncate message 5, and every cached value from position 5 onward is gone. If you need to manage context length, summarize from the beginning. It invalidates the cache once, but the compressed conversation caches fresh and subsequent turns benefit.

Warm the cache before going parallel. A cache entry only becomes available after the first response begins. If you fire 10 concurrent requests, only the first one writes the cache. The rest all miss. Send one request, wait for the first token, then fan out.

Use the 1-hour TTL for slow agents. Anthropic's default cache TTL is 5 minutes. If your agent takes longer than that between steps (waiting on human approval, running a slow tool, hitting rate limits), the cache expires. The 1-hour TTL costs 2x base input price to write, but prevents full recomputation on every delayed step.

Silent Cache Killers

Beyond the obvious invalidators (tool changes, system prompt edits), these break your cache with no error and no warning:

• Toggling image attachments on or off between requests
• Changing tool_choice between turns
• Enabling or disabling extended thinking
• Non-deterministic key ordering in tool_use content blocks (common in Swift and Go)
• Switching between workspaces. On the Claude API, caches are isolated per workspace as of Feb 2026, so a request hitting workspace B will not see workspace A's cached prefix.
• Upgrading model versions. A new model version can change how text is tokenized, which changes the token sequence and silently breaks any prefix cached under the old version. Re-warm the cache after a version bump.

Monitoring

Every Anthropic response reports the same usage fields we saw earlier:

{
  "cache_read_input_tokens": 98432,    // should be high after turn 1
  "cache_creation_input_tokens": 0,    // should be 0 after turn 1
  "input_tokens": 47                   // only the new, uncached part
}

If cache_read_input_tokens stays at 0, something is breaking your prefix. Check: timestamps in the system prompt, non-deterministic serialization, tool set changes, or gaps exceeding the TTL.

The Economics

Quick math for an agent workflow (30 round trips, 50K-token prefix, Claude Sonnet 4.6 at $3/MTok input and $0.30/MTok cache read):

The blended saving lands at 86% rather than the full 90% read discount because the first call pays the 1.25x write surcharge. After that one write, every hit is 90% off: by the second request you're saving money, and by the 30th it's not close.

Prompt caching is what makes agents economically viable. Without it, every round trip reruns full inference on your entire prompt; tool-calling workflows with large system prompts would cost 5-10x more per task. The cache is the load-bearing wall, not a tuning knob you visit later.

Cache Optimization Checklist

Prompt structure:
  [ ] Stable content first: tools → system prompt → context → history → new message
  [ ] No timestamps, session IDs, or mode flags in the system prompt
  [ ] Dynamic state lives in messages array, not system

Serialization:
  [ ] json.dumps(result, sort_keys=True) on all tool outputs
  [ ] Stable key ordering in tool_use content blocks (watch Go/Swift)

Tool management:
  [ ] Same tool set in every request (no adding/removing mid-conversation)
  [ ] Behavior changes via system messages, not tool-set swaps

Conversation management:
  [ ] History is append-only (never edit or truncate earlier messages)
  [ ] Context compression happens at the start, not the middle

Cache hygiene:
  [ ] Warm the cache with one request before fanning out parallel calls
  [ ] Use 1-hour TTL if agent steps take >5 minutes
  [ ] No toggling of image attachments or extended thinking between turns
  [ ] tool_choice stays consistent across the conversation
  [ ] Same workspace on every request (Anthropic isolates caches per workspace)
  [ ] Pin the model version; tokenizer changes between versions break the cache

Monitoring:
  [ ] cache_read_input_tokens is high after turn 1
  [ ] cache_creation_input_tokens is 0 after turn 1
  [ ] If both are 0, debug: timestamps? serialization? tool changes? TTL expiry?

---

Mechanics coming in a follow-up. The rules above are enough to apply prompt caching well. If you want to understand why they hold (transformer layers, attention math, the full prefill vs decode internals), I'm writing that as a separate post next.

References

• Anthropic Prompt Caching Docs
• Anthropic: What's New in Claude 4.7
• Anthropic Release Notes
• OpenAI Prompt Caching Guide
• Gemini Implicit Caching Announcement