Prompt caching in practice: breakpoints, TTLs and cost math

TL;DR

• Prompt caching bills a repeated prompt prefix at 2 to 10 percent of the input rate on current Anthropic, OpenAI and DeepSeek models, but only when the prefix is byte-identical, long enough and still in the cache.
• Anthropic and OpenAI’s GPT-5.6 family charge 1.25 times input to write a short-lived entry; Gemini’s explicit caches charge storage by the hour instead; DeepSeek and older OpenAI models charge nothing extra to write.
• In a 40-turn agent loop priced below, caching cuts the input-plus-output bill by 81 to 91 percent across providers; a single timestamp re-rendered into the system prompt each turn turns that into a 24 percent increase.
• Hit rate comes from the usage fields, and the formulas differ: Anthropic and Bedrock report input_tokens net of cache activity, OpenAI reports it gross.
• Gateways decide whether a cache stays warm. Bifrost, an open-source AI gateway, pins sessions to the provider key that holds their cache and injects breakpoints for clients that send none.

Prompt caching is a provider-side discount on repeated input: when a request begins with the same tokens as a recent one, the model reuses the work it already did on that prefix and bills those tokens at a fraction of the normal rate. Every major model API offers it, but each one sets its own rules for markers, minimum lengths, lifetimes and write prices. This guide covers the implementation side of prompt caching provider by provider, from the request fields to the cost arithmetic.

How prompt caching works

Prompt caching stores the attention key-value (KV) state a model computes for a prompt prefix and loads it when a later request starts with an identical prefix. The model skips recomputing those tokens and processes only what follows, so input gets cheaper and time to first token drops. Output is generated and billed exactly as before.

OpenAI’s guide puts the mechanism plainly: the prompt cache stores key-value tensors, not the tokens themselves. Self-hosted inference servers do the same thing in block form. vLLM’s automatic prefix caching hashes each block of tokens together with every token before it, so a block is only reusable when its whole prefix matches. The Frontier Wire’s comparison of prompt caching and semantic caching covers that mechanism and how it differs from response caching in more depth; this piece stays with the practical question of making provider caches hit.

Three consequences follow from the mechanism and shape everything below:

• Order matters. A cache is a prefix, so content that changes must come after content that does not. One changed token early in the prompt invalidates everything after it.
• Placement matters. Providers write cache entries only at specific boundaries, called breakpoints. A breakpoint placed on a block that changes every request produces a new write every time and no reads.
• Time matters. Entries expire. Anthropic’s 5-minute default is measured from the start of the request that writes or reads the entry, so a response that streams for four minutes leaves about one minute for the follow-up.

Prompt caching by provider at a glance

The five largest API providers split into two camps: those that require a marker or offer one (Anthropic, OpenAI on GPT-5.6 and later, Bedrock, Gemini’s explicit caches) and those that cache automatically with no request changes (OpenAI’s earlier models, Gemini’s implicit caching, DeepSeek). The table summarizes the current rules from each provider’s own documentation.

The rest of this guide takes each row in turn, then covers prompt structure, measurement, cost and the gateway layer. List prices for the underlying models are compared across providers in the Frontier Wire’s LLM API pricing comparison.

Anthropic prompt caching with cache_control

Claude prompt caching is turned on with one field, cache_control, placed either at the top level of the request or on individual content blocks. Anthropic’s prompt caching documentation states that caching is supported on all active Claude models, with cached prefixes built in a fixed order: tools, then system, then messages.

Automatic caching is the simpler mode. A single top-level cache_control places the breakpoint on the last cacheable block and moves it forward as the conversation grows, so each turn reads everything up to the previous turn and writes only the new tail. It uses one of the four breakpoint slots, and the API returns a 400 error if four explicit block-level breakpoints already exist.

Explicit breakpoints put cache_control on specific blocks, up to four per request. Their behaviour has a detail that causes most misconfigured Claude caches. Anthropic’s section on how automatic prefix checking works sets out three rules:

1. Cache writes happen only at a breakpoint. The system does not write entries for earlier positions.
2. On a miss at the breakpoint, the system walks backward looking for entries that earlier requests wrote, not for stable content.
3. The lookback covers at most 20 blocks per breakpoint.

The documentation’s own example of the common mistake is a breakpoint on a block that contains a timestamp and the user message. Every request writes a new entry, the lookback finds no earlier writes at the stable blocks, and the application pays the write premium on every call while never reading. The fix is to move the breakpoint to the last block that stays identical across requests.

Pricing for the current models, from Anthropic’s prompt caching price table, per million tokens:

Several less obvious rules from the same documentation affect production systems:

• The minimum is silent. The minimum cacheable length is 512 tokens on Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1, 2,048 on Opus 4.7, and 4,096 on Opus 4.6, Opus 4.5 and Haiku 4.5. A shorter prompt is processed without caching and no error is returned.
• Parallel requests miss. For concurrent requests, a cache entry only becomes available after the first response begins, so a fan-out of identical calls should wait for the first one to start streaming.
• Isolation is per workspace. Caches are isolated per workspace on the Claude API, while Bedrock and Google Cloud isolate at the organization level.
• Settings invalidate. Changing tool definitions invalidates the whole cache; toggling web search or citations invalidates system and messages; changing tool_choice, images, thinking parameters or the effort setting invalidates the messages level. The full matrix is under what invalidates the cache.
• Pre-warming is supported. A request with max_tokens: 0 writes the cache and returns no output, so an application can load its system prompt before the first user arrives. It pays the normal write charge and no output.

The 1-hour lifetime is meant for prefixes reused less often than every 5 minutes but more often than hourly, and Anthropic notes that cache hits are not deducted against rate limits. When both lifetimes appear in one request, 1-hour entries must come before 5-minute ones. Teams paying for Claude through a subscription rather than the API face different economics, covered in the Frontier Wire’s breakdown of Claude Code pricing.

OpenAI prompt caching and prompt_cache_key

OpenAI prompt caching is on by default and needs no code changes, but GPT-5.6 changed how it works. On GPT-5.6 and later models, developers can mark explicit breakpoints and choose a lifetime, and cache writes now carry a surcharge; on earlier models caching is implicit only and writes are free.

For GPT-5.6 and later, the guide’s section on how caching works describes two modes set through prompt_cache_options.mode:

• Implicit mode (the default) places a breakpoint at the end of the latest eligible message: a user message, the last tool response in a run of tool responses, or the last developer message in the opening block. Developers can add explicit breakpoints alongside it.
• Explicit-only mode uses only breakpoints marked with prompt_cache_breakpoint on a content block. Content after the last breakpoint is billed at the ordinary input rate with no write charge, which avoids paying to write a changing suffix that will never be read.

Each request can create up to four cache writes. The cache lifetime is set with prompt_cache_options.ttl: "30m" by default at 1.25 times the uncached input rate, or "12h" on the Responses API at 2.5 times. The guide stresses that cache-write pricing is not an additive fee: each input token is billed once, at the uncached, cached or write rate.

Current prices from OpenAI’s pricing page, per million tokens at standard short-context rates:

The prompt_cache_key parameter has two different jobs depending on the model. On models before GPT-5.6, it influences routing: OpenAI advises a stable key for requests that share a prefix and, for busy groups, about 15 requests per minute per key, splitting heavier traffic across several keys deterministically. That figure reflects how caches are stored: cached states live on individual machines, and traffic above that rate can overflow to machines without the entry. On GPT-5.6 and later, routing is automatic and the key is optional, used to keep separate cache accounting per customer and to prevent cache-hit probing across them.

OpenAI’s gotchas section documents failures that catch teams migrating to GPT-5.6. In implicit mode, a request whose static developer message is followed by a dynamic user message writes through the dynamic content, so the next request cannot reuse the shorter shared prefix; the remedy is an explicit breakpoint after the static content. Switching a conversation from implicit to explicit-only mode can also miss the previous implicit write. On rate limits, OpenAI’s answer is unambiguous: cached input tokens still count toward tokens-per-minute limits.

Gemini context caching: implicit and explicit

Gemini offers two mechanisms. Implicit caching is on by default for Gemini 2.5 and newer and passes on savings automatically but, in Google’s words, carries no cost saving guarantee. Explicit context caching creates a named cache object that requests reference, guarantees the discount and adds an hourly storage charge.

The implicit caching minimum is 4,096 tokens on Gemini 4 Argon, 3.8 Flash, 3.6 Flash and 3.1 Pro Preview, and 2,048 on Gemini 2.5 Flash and 2.5 Pro. Google’s advice for raising the hit rate is the same as everyone else’s: put large, common content at the start of the prompt and send requests with a similar prefix close together.

Explicit caching is in beta under v1beta, works only through the generateContent API, and is not supported on Gemini 4 Argon and later, which use implicit caching. A cache is created with caches.create, carrying a system instruction and contents such as an uploaded PDF or video, and later requests pass its name as cached_content. The TTL defaults to 1 hour, and the documentation states that there are no minimum or maximum bounds on the TTL. Standard rate limits still apply and token limits include cached tokens.

Prices from Gemini’s pricing page, per million tokens on the paid tier:

The storage line changes the decision. On Gemini, an explicit cache is worth creating only when the per-query saving exceeds the hourly storage, which is a function of query rate, not just prefix size. The cost section below works through one case.

Bedrock prompt caching across Claude, Nova and GPT

Amazon Bedrock supports both implicit and explicit prompt caching, with support varying by model and API. Its documentation is direct that implicit prompt caching is best effort, and that support for caching never guarantees a hit, which is why it tells developers to check the usage fields on every response.

Explicit caching on Bedrock uses a cachePoint block in the Converse API, or cache_control in an InvokeModel body for Claude. The supported models table lists the per-model rules: Claude 5.x models need 512 tokens per checkpoint, take up to four, and support 5-minute and 1-hour lifetimes; Opus 4.7 and Haiku 4.5 need 4,096; Amazon Nova models need about 1,000 tokens, support 5 minutes only and cache at most 20,000 tokens. The minimum applies cumulatively to everything before each checkpoint, including tools and system content.

Bedrock also offers Anthropic-style simplified cache management: one checkpoint at the end of the static content, with a lookback of about 20 content blocks for an earlier match.

OpenAI’s GPT-5.6 and later models on Bedrock follow OpenAI’s own rules: implicit and explicit modes, a 1,024-token minimum, a 30-minute minimum lifetime and writes at 1.25 times input. One rule differs from OpenAI’s own API. Bedrock states for these models that cached input tokens do not count against the input-tokens-per-minute quota, while OpenAI counts them. Three further Bedrock constraints matter in design:

• No batch. Prompt caching is only supported for on-demand inference endpoints, not the batch inference API.
• Cross-region costs writes. With cross-region inference, these optimizations may lead to increased cache writes at times of high demand.
• Field names differ. Converse reports cacheReadInputTokens and cacheWriteInputTokens, and its inputTokens field counts only non-cached tokens.

DeepSeek and self-hosted prefix caching

DeepSeek’s context caching is on by default, stores prefixes on disk rather than in GPU memory, and charges nothing extra to write. It also has the steepest read discount of the major APIs, which makes cache-friendly prompt structure worth more there than anywhere else.

DeepSeek’s documentation describes how cache prefixes are persisted: at the end of the user input and the end of the model output of each request, at fixed token intervals for long inputs, and when the system detects a common prefix shared by several requests. A hit requires a full match of one of these persisted units. In DeepSeek’s long-document example, the first two questions about the same report miss, the system then persists the shared report as a unit, and the third question hits. The service is best effort, and cache construction takes seconds, with unused entries cleared within hours to days.

The usage fields are prompt_cache_hit_tokens and prompt_cache_miss_tokens. On the pricing page, deepseek-flash costs $0.30 per million input tokens on a miss and $0.006 on a hit at peak, and deepseek-v4-pro costs $1.32 and $0.044; off-peak rates are half of the peak rates.

Teams that serve open-weight models themselves get the same behaviour from the inference engine. vLLM’s prefix cache is per server, so a router in front of several replicas decides whether consecutive turns land where their blocks are; the Frontier Wire survey of routers for vLLM, Ollama and cloud models covers that layer. vLLM also supports isolating prefix cache reuse through optional per-request salting, so tenants that do not share a salt never share blocks.

Structuring prompts for cache hits

A cache-friendly prompt puts everything that is identical across requests first, in a deterministic order, and everything that varies last. The breakpoint goes on the last stable block, not on the block that changes. Every provider’s documentation gives the same advice in different words, because the cache is a prefix hash.

Figure 1: Everything before a breakpoint must be byte-identical across requests; anything that changes per request belongs after the last one.

As Figure 1 shows, a typical layout has two breakpoints: one after the static prefix of tools, instructions and reference material, and one at the end of the conversation history, which moves forward each turn. The rules that keep that layout stable:

• Keep volatile values out of the prefix. Dates, request IDs, user names and feature flags belong in the final message, or in a later developer or system message. Anthropic supports appending a mid-conversation {"role": "system"} message on newer models instead of editing the top-level system field, and OpenAI recommends placing dynamic developer content after the breakpoint.
• Serialize deterministically. Anthropic’s troubleshooting list warns that some languages randomize JSON key order, naming Swift and Go, which silently breaks caches on tool-use blocks. Sort keys and keep tool order fixed.
• Keep tools stable. OpenAI’s guidance on append-only tool updates is to keep definitions and order constant, use tool_choice: "none" or allowed_tools to restrict calls, and load tools late with tool search so new ones are appended rather than inserted.
• Append, do not rewrite. Summarizing or truncating history changes the prefix. OpenAI notes that compaction can lower the hit rate while still lowering total cost, so the comparison should be on total input spend.
• Hold settings constant. Reasoning effort, thinking budgets, structured output schemas and tool_choice are rendered into the prompt on several providers. Changing them mid-conversation invalidates the message-level cache on Claude, and OpenAI’s table of settings that affect the cached prefix lists the same family of parameters.
• Clear the minimum. OpenAI documents a minimum cacheable length cost trap: with a 1,024-token minimum, 0.1 times reads and 1.25 times writes, the break-even prefix length is 102.4 plus 1,177.6 divided by the number of requests. Across 10 requests, padding any prefix of at least 221 tokens up to 1,024 with useful stable material is cheaper than leaving it uncached.

Measuring cache hit rate from usage fields

The cache hit rate that matters for cost is the token hit rate: cached input tokens divided by total input tokens, aggregated by route, prefix or customer. Every provider reports the components in the response, but they define input_tokens differently, so the formula has to be adapted per provider before numbers can be compared.

Figure 2: Anthropic and Bedrock report input_tokens net of cache activity while OpenAI reports it gross, so the formulas differ before the metrics can be compared.

Anthropic’s tracking cache performance section is explicit that input_tokens covers only tokens after the last breakpoint, so a dashboard that reads it as total input will understate traffic by orders of magnitude on a well-cached workload. OpenAI’s monitoring guidance takes the opposite convention: ordinary input is input_tokens minus cached_tokens minus cache_write_tokens. Its worked example at $1 input and $0.10 cached input prices 1,000 ordinary, 4,000 cached, 3,000 30-minute and 4,000 12-hour tokens at $0.01515.

Two more signals are worth tracking alongside the hit rate:

• Write-to-read ratio per prefix. A healthy agent loop writes once and reads many times. If every turn reports writes and no reads, something in the prefix changes each turn. Anthropic’s diagnostic shortcut is that if both cache fields are zero, the prompt was most likely under the minimum.
• Realized input cost. Multiply each count by its rate and compare with the uncached cost of the same tokens. This catches a cache that hits often but writes expensive 1-hour or 12-hour entries that are rarely read.

Both Anthropic and OpenAI also offer tooling that compares consecutive requests and reports where a prefix diverged: Anthropic’s cache diagnostics and OpenAI’s Prompt Cache Diagnostics tool.

Prompt caching cost math with current prices

Prompt caching pays off when a prefix is read more often than it is written, and the margin depends on the write surcharge and read discount. Anthropic’s pricing page states that a 5-minute cache pays off after one cache read, and a 1-hour cache after two. Three worked examples follow, using list prices fetched on October 21, 2026.

An agent loop across five providers

The workload is a 40-turn coding agent. The first request carries a 12,000-token static prefix of tools and instructions. Each later turn appends 2,500 tokens (a 500-token model reply and a 2,000-token tool result), and each turn generates 500 output tokens. Total input over the run is 2,430,000 tokens and output is 20,000. With caching, the first turn writes 12,000 tokens, each later turn writes its 2,500 new tokens and reads everything before them: 109,500 tokens written and 2,320,500 read, a token hit rate of 95.5 percent.

On Sonnet 5.5 the cached run breaks down to $0.27 of writes, $0.23 of reads and $0.20 of output. The Gemini and DeepSeek rows assume their implicit caches hit as the explicit ones do, which neither provider guarantees, so they are best cases. Output becomes a large share of the remaining bill once the input is cached, which is where response caching and the reasoning-token issues discussed in the hub article on prompt and semantic caching take over.

The same run with a timestamp re-rendered into the system prompt on every turn shows the downside. Every turn misses, the full prompt is written at $2.50 per million on Sonnet 5.5, and the run costs $6.28: 24 percent more than not caching at all.

Choosing between 5-minute and 1-hour entries

A support conversation reuses a 20,000-token prefix over 10 turns, with the user replying about every 10 minutes. On Sonnet 5.5 without caching, the prefix costs $0.40 across the conversation. With the 5-minute default, every turn arrives after the entry has expired, so all 10 turns write at $2.50 per million and the prefix costs $0.50, more than no caching. With a 1-hour entry, one write at $4 per million and nine reads at $0.10 cost $0.098, a 76 percent saving. The rule is to match the lifetime to the gap between requests, not to the length of the session.

Gemini explicit storage against query rate

A team keeps a 200,000-token document corpus in an explicit cache on Gemini 3.8 Flash. Storage costs $0.50 per million tokens per hour, or $0.10 an hour for this cache. Each query that reads it pays $0.015 for the cached tokens instead of $0.15 uncached, saving $0.135. The cache pays for its storage at about 0.74 queries an hour; at 20 queries an hour over an 8-hour day, $0.80 of storage saves $21.60 of input. These figures exclude the request that creates the cache. At low query rates, the implicit cache, with no storage line, is the better default.

Common prompt caching pitfalls

Most low hit rates come from a small set of causes: a prefix that is not actually stable, a breakpoint in the wrong place, entries that expire between requests, or traffic that lands somewhere the cache does not live. The list below collects the ones documented by the providers themselves.

• Breakpoint on the changing block. On Claude, writes only happen at breakpoints, so a marker on the user message writes a new entry each request and never reads one.
• Falling out of the lookback window. A Claude conversation that adds 20 or more blocks in one turn moves the breakpoint past the last write. A second breakpoint closer to the previous position fixes it.
• Parallel first requests. A burst of identical calls all miss and all write, because the entry is not readable until the first response starts.
• Model or provider switches. Caches are per model and per provider account. Every failover to another provider, and every change of model within a session, starts cold, which is worth factoring into failover and load-balancing policies.
• Spreading one session across keys or workspaces. Anthropic isolates caches per workspace, OpenAI per organization and region. A load balancer that rotates a conversation across several API keys in different organizations or workspaces discards its cache each time.
• Prompt-length pricing. On Claude Haiku 5.5, which is priced by prompt length, cache reads and writes count toward a request’s prompt length, so a request over 100,000 tokens pays the higher rate even when most of it is a cache hit.
• Rate-limit assumptions. Cached tokens count against tokens-per-minute limits on OpenAI’s API but not for Claude on the Claude API or for GPT-5.6 models on Bedrock. Capacity planning built on one provider’s rule breaks on another; the Frontier Wire’s explainer on rate limiting covers how those quotas are enforced.

Caching also has a privacy dimension. A 2025 ICML paper, Auditing Prompt Caching in Language Model APIs, measured cache-induced timing differences and reported detecting global cache sharing across users at seven API providers at the time of the audit, including OpenAI. OpenAI’s current documentation states that caches are not shared across organizations, and Anthropic’s isolates by workspace. Within one account, though, the boundary is the credential: LiteLLM’s documentation notes that the provider caches a prefix against the upstream credentials, so tenants sharing one key can infer from hit timing whether a prompt they already hold was recently sent. Separate credentials, OpenAI’s per-customer prompt_cache_key or vLLM’s cache salt are the controls.

Prompt caching through an AI gateway

An AI gateway sits between applications and providers, so it can either preserve or break prompt caching. It breaks caching when it rotates a session across keys, rewrites earlier messages or strips markers. It helps when it keeps sessions on one key, adds markers for clients that send none, and prices cache reads and writes correctly.

Bifrost is used here as the worked example because its documentation covers all three behaviours in detail. Bifrost is an open-source AI gateway written in Go, released by Maxim AI under the Apache 2.0 license, that exposes 25+ providers and 10,000+ models through one OpenAI-compatible API. Its published benchmark reports 11 µs of overhead at 5,000 RPS on an AWS t3.xlarge.

Figure 3: Affinity keeps each turn on the provider key that holds the cache; injection gives marker-less clients a breakpoint in each provider’s own format.

Session affinity. Bifrost’s session affinity keeps a conversation or agent run on the provider and key that served it before, because provider prompt caches are per provider and usually per key or organization. A session is identified by an x-bf-session-id header, and coding harnesses such as Claude Code, Codex CLI and OpenCode get affinity with no configuration because the gateway adopts the session headers they already send. Routing still decides which providers and keys are allowed; affinity only reorders them, and a binding expires after one hour of inactivity by default. The documentation’s limits include that a new session’s first parallel burst can scatter once before a binding exists.

Marker injection. The auto prompt caching setting addresses agentic clients that send no cache markers, naming Codex as one; on Anthropic models such a client caches nothing. With prompt_cache.auto_inject enabled on a provider, the gateway marks the first cacheable block of requests that arrive without markers. The marker is translated per provider, as the provider support table lists: cache_control for Claude on Anthropic and Vertex AI, cachePoint on Bedrock, and prompt_cache_breakpoint for OpenAI’s GPT-5.6 family; Gemini, DeepSeek and other implicit-caching providers are left alone.

Injection is off by default, caller markers always take precedence, and at most four markers are added. Injection points can target messages by role and index instead, mirroring LiteLLM’s setting of the same name. A per-request header turns injection on or off for one call on providers where the setting is configured.

Measurement and cost. The gateway surfaces each provider’s counters as cached_read_tokens and cached_write_tokens under usage.prompt_tokens_details, and its verification guidance matches the diagnosis above: writes on every turn and no reads mean the prefix is changing. The model catalog prices cache-read and cache-write tokens at separate rates. The documentation’s own cost considerations warn that injecting a marker on one-shot traffic adds 25 to 100 percent to the marked prefix for nothing, so injection belongs on providers that serve agent loops and multi-turn chat.

Enterprise tier. Health-aware invalidation of session bindings and cluster-wide replication of bindings are part of Bifrost Enterprise.

Other gateways cover parts of the same ground. LiteLLM documents cache_control_injection_points and an environment flag that adds checkpoints for Claude automatically, and it warns that adding injection points to a Gemini deployment switches it to explicit context caching.

Further reading on the gateway side includes a survey of AI gateways with prompt caching support, a practical guide to prompt caching and a comparison of gateways that combine caching with smart routing.

The Bifrost team’s post on LLM cost optimization at the gateway places prompt caching alongside semantic caching, tool-context reduction and routing, and an article on cutting cost and latency with semantic caching covers the response-cache layer that sits in front. Separately, the Frontier Wire’s guide to LLM cost control at the gateway sets caching next to budgets and routing as one of several controls.

Where the savings come from

Prompt caching is now the cheapest large saving available on most model APIs, and it is mostly prompt hygiene: stable content first, breakpoints after it, volatile values last, settings held constant and lifetimes matched to the gap between requests. On the agent workload priced above, that hygiene is worth more than 80 percent of the bill on every provider tested on paper, and its absence can make caching cost more than leaving it off.

The provider differences are where implementations go wrong. Write surcharges exist on Anthropic and on OpenAI’s newest models but not on DeepSeek or OpenAI’s older ones; Gemini replaces them with storage; usage fields count input in opposite ways; and cached tokens count against rate limits on some platforms and not others. Teams running several providers should normalize the usage fields into one hit rate and one realized-cost figure per route before trusting any dashboard. The limits of this analysis are the ones the providers state themselves: implicit caches on Gemini, Bedrock and DeepSeek are best effort, and none of the worked examples above measures real hit rates, which only production traffic can show.