What Is Prompt Caching? How It Works and When It Saves Money

Suppose an internal policy Q&A bot sends the same 5,000-token system prompt on every call: the HR handbook excerpt, the expense rules, and the formatting instructions. The employee's question that follows is usually about 40 words. Without prompt caching, you pay full input price for that handbook excerpt thousands of times a day, and the model reprocesses the same text each time. Ramp's write-up on AI token spend describes finding avoidable spend tied to exactly this kind of missing caching. The fix is usually a change to the order of the prompt plus one configuration flag.

This guide covers what prompt caching is, how it works, how to structure prompts so it works, and how to check whether your current bill is getting the discount.

What Prompt Caching Is (and What It Is Not)

A language model handles a request in two phases. In prefill, it reads the entire prompt and builds an internal representation of every token, called the key-value (KV) cache. In decode, it generates the response one token at a time from that representation. Prefill on a long prompt is expensive, and most of the input-token charge pays for it.

Prompt caching stores the KV cache for a prompt prefix after the first request. When a later request starts with exactly the same tokens, the provider loads the stored state, runs prefill only on the new part, and bills the reused tokens at a cached-input rate. Morph's provider comparison puts the typical cached rate at 0.1x base input. The same comparison lists 0.025x for Claude Fable 5.1, 0.032x for DeepSeek V4 during off-peak hours, and $0.1955 against $1.19 per million tokens for GLM-5.3 744B on Morph, which is about 84% off. The output is unchanged because the model computes the same thing. It just doesn't compute it twice.

Three different techniques share the word "caching," and people often mix them up in budget discussions:

Semantic caching skips the model call completely when a new question resembles an old one, so it carries a correctness risk. We cover the tooling in Best Libraries for Caching in an Agent Knowledge Base. Plan caching reuses an agent's reasoning about how to approach a task, which is covered in How to Reduce AI Agent Cost With Plan Caching. Prompt caching is the only one of the three that has no effect on quality. That makes it the right place to start, since the only downside is a small amount of engineering discipline.

How the Cache Matches: Prefixes, Not Fragments

Providers match the cache from the first token forward. Once a single token differs, everything after it is a miss, even if the remaining text is identical to a cached request. That rule is behind nearly every caching mistake.

Here are two layouts for the same policy bot:

Layout A (cache-hostile)

1. "Today is 2026-10-22 14:03:17. You are helping Priya in Finance."
2. 5,000-token policy handbook
3. Formatting rules
4. User question

Layout B (cache-friendly)

1. Formatting rules and role instructions
2. 5,000-token policy handbook
3. "Today is 2026-10-22. User: Priya, Finance."
4. User question

Layout A misses on every call because the timestamp and user name at the top change on every request. Layout B carries the same information, but the first ~5,000 tokens are byte-identical across all users, so the provider can serve them from cache.

Order the prompt from most stable to least stable:

• Tier 1, static for weeks: role instructions, output format, tool definitions, few-shot examples
• Tier 2, static for hours or days: reference documents, retrieved policy text, product catalog snapshot
• Tier 3, static within a session: conversation history so far
• Tier 4, changes every call: the new user message, current time, request metadata

Providers implement this differently. OpenAI's prompt caching guide applies caching automatically once a prompt passes a minimum length, so the main work there is getting the order right. Anthropic's API asks you to mark cache breakpoints explicitly with cache_control, and a cache write costs more than a normal input token. On Anthropic, a prefix that is written once and never read again costs you more than not caching it. Google's Gemini context caching docs describe both implicit caching and an explicit cache object with its own lifetime that you create and reference.

Guides such as Digital Applied's engineering walkthrough cover the per-provider configuration. Whichever provider you use, the stable content goes first.

When It Saves Money (and When It Doesn't)

Two numbers drive the savings: how much of each prompt is a reusable prefix, and how often requests arrive within the retention window.

Here is an illustrative calculation for the policy bot. These are not real prices or client results. Substitute your own rates.

• Base input price: $3.00 per million tokens (illustrative)
• Cached input rate: 0.1x, so $0.30 per million
• Stable prefix: 5,000 tokens
• Variable suffix (question plus metadata): 200 tokens
• Volume: 10,000 calls per day, 30 days

That is $4,050 a month on one internal bot, and the gap grows with traffic. The table assumes every call hits, which is too optimistic. The first request after a cache expiry pays full price, plus the write premium on providers that charge one. At 10,000 calls a day spread across working hours, the cache stays warm almost all the time. At 50 calls a day, it barely helps.

Cost savings are the main argument for prompt caching, but latency improves too. Skipping prefill on 5,000 tokens shortens time to first token, which users notice in a chat interface.

Caching does very little in these cases:

• Short prompts. Below the provider's minimum cacheable length, nothing is cached.
• Sparse traffic. A nightly batch job that runs once per document type never reuses a warm cache.
• Prompts that are mostly unique. A summarizer that receives a different 20-page contract on every call has very little shared prefix. Batch APIs or a smaller model will save more there.
• Output-heavy workloads. Caching discounts input only. If most of your spend is output tokens, as with long code generation, the input discount is a small share of the bill. Know Your Token Mix explains how to find out which case you're in.

Agents get the most benefit. Every turn of an agent loop resends the system prompt, the tool definitions, and the growing transcript. Without caching, a 30-turn session pays for the opening tokens 30 times. With caching and correct ordering, each turn pays full price only for the new tool result and the new model message. This also affects the math in Long Context vs RAG Cost. A large, stable context that is cached can cost less than a retrieval pipeline that changes the prompt on every call.

What Breaks the Cache

Most teams that "have caching turned on" still miss more often than they expect. The causes tend to be small:

1. Dynamic values near the top. Timestamps, request IDs, user names, A/B test flags, and "current date" lines in the system prompt. Move them below the stable block, or reduce them to a coarser value (date instead of datetime) if they need to appear early.
2. Non-deterministic serialization. If tool schemas come from a dictionary whose key order varies, or a plugin system loads tools in arbitrary order, the bytes change even though the meaning is the same. Sort tool definitions and JSON keys explicitly.
3. Changing the tool list per request. Adding or removing one tool based on the user's role changes everything after the tool block. Keep one stable superset of tools and enforce permissions at execution time.
4. Idle sessions. Default retention windows are measured in minutes. A user who reads a long answer for a while and then asks a follow-up may come back to a cold cache. Some providers sell longer retention, but you pay for it.
5. Editing history. Context compaction, summarizing old turns, or trimming tool output in place rewrites the middle of the prompt and invalidates everything after the edit. Compaction is often still worth doing (see Context Compaction for Agents), but do it rarely and at clear boundaries rather than on every turn.
6. Model or routing changes. Caches are tied to a specific model. A router that switches between two models on alternating turns warms neither cache.
7. Provider-side changes. Caching runs inside the provider's serving stack, which you don't control. Anthropic's April 23 engineering postmortem is a useful example of how cache behavior interacts with production serving systems and where its limits are. Track your hit rate as a metric instead of assuming it stays where it was.

Large prompts cause their own trouble. A very long stable prefix is cheap to cache, but it still uses context window and attention. Shopify's engineering post on gisting covers the related problem of long, repeated prompt content. Caching lowers the price of a big prefix, but you should still ask whether the prefix needs to be that big.

How to Check Whether You're Using It

You can get an answer within an hour. Work through three sources, from cheapest to most detailed.

1. The invoice or billing dashboard. Look for a separate line item or column for cached input tokens. If your provider breaks out cached reads and you see zero, or nothing next to a large input-token total, caching isn't happening. Finance teams can usually run this check without involving engineering. Flexera's breakdown frames cached tokens as a FinOps line item, which is a useful way to bring it up with the people who own the budget.

2. The usage object on API responses. Each provider reports cache activity per request:

• OpenAI: usage.prompt_tokens_details.cached_tokens
• Anthropic: usage.cache_read_input_tokens (hits), usage.cache_creation_input_tokens (writes), usage.input_tokens (uncached remainder)
• Gemini: usage_metadata.cached_content_token_count

3. A hit-rate metric in your logs. Record these fields on every call and compute a ratio per workload. A minimal version for Anthropic-style usage:

def cache_hit_rate(usages):
    read = sum(u["cache_read_input_tokens"] for u in usages)
    write = sum(u["cache_creation_input_tokens"] for u in usages)
    fresh = sum(u["input_tokens"] for u in usages)
    total = read + write + fresh
    return read / total if total else 0.0

Calculate it per endpoint or per agent rather than across the whole account, because a single average hides the one workload sending 5,000 uncached tokens per call. To interpret the result:

If a workload with a stable prefix shows a low hit rate, check the top of the prompt first. Log the first 500 characters of two consecutive requests and diff them. In most cases, the diff shows a timestamp or a reordered tool.

Rolling Prompt Caching Out Across an Agent Fleet

Fixing one bot takes an afternoon. Keeping caching working across a dozen agents owned by different teams is mostly a governance problem: prompt templates drift, someone adds a "current time" line to a shared system prompt, and the hit rate drops across the board with nobody alerted.

A workable rollout looks like this:

1. Inventory. List every LLM-calling workload with its daily input tokens, prefix size, and current hit rate. Rank them by uncached prefix tokens per day. The top three usually account for most of the possible savings.
2. Restructure. Reorder those prompts into stable-to-variable tiers, make serialization deterministic, and add explicit breakpoints where the provider requires them.
3. Guard. Add a test that renders the prompt for two different users and asserts that the first N tokens are identical. This catches regressions at review time instead of on the invoice.
4. Monitor. Put hit rate per workload on the same dashboard as latency and error rate, with an alert when it drops. Treat it as an operational metric, and don't rely on a one-time configuration.

OpenNash does this as part of cost engineering for production agents: we audit token flows, restructure prompts and harnesses so caching holds up, and add the regression tests and dashboards so your team can keep the savings after we hand the system over. If your AI bill is growing faster than usage and nobody can say what share of input tokens is cached, book a token-spend review with OpenNash and we'll go through your highest-volume workloads together.

If you'd rather do it yourself, start this week. Pull yesterday's usage records for your highest-volume endpoint, calculate the cached share of input tokens, and diff the first 500 characters of two requests. If that endpoint has a large fixed prefix and a hit rate near zero, reorder the prompt before you spend time on any other cost project.