Prompt caching vs semantic caching: what each one saves

TL;DR

• Prompt caching is a provider feature: the model still runs, but a repeated prompt prefix is billed at 5 to 10 percent of the input rate on current Anthropic and OpenAI models. Output tokens are never discounted.
• Semantic caching sits in front of the provider and returns a stored response when a new request is close enough in meaning, so a hit removes the whole call, output included.
• Prompt caching cannot change an answer. Semantic caching can, and research on verified caches shows a single static similarity threshold gives no guarantee on error rate.
• The two stack. In the worked example below, prompt caching cuts a support assistant’s daily bill by about 86 percent, and a semantic cache with a modest hit rate then removes a further share that is mostly output tokens.
• Bifrost, an open-source AI gateway, runs exact-match and semantic response caching and can also insert provider prompt-cache markers for clients that send none.

Prompt caching and semantic caching both promise lower LLM bills, but they cache different things in different places. Prompt caching lets the model provider reuse the computation for a prompt prefix it has already seen, so the request still runs and the input is discounted. Semantic caching stores finished responses before the provider and replays one when a new request means roughly the same thing, so the request never reaches the model at all. Choosing between them, or running both, depends on where a workload repeats itself: in its context, or in its questions.

Prompt caching and semantic caching compared

Prompt caching reuses the processed form of an identical prompt prefix inside the provider and bills those input tokens at a discount. Semantic caching reuses a complete stored response for a request that is similar in meaning and skips the provider. The table sets out the differences that decide which one a workload needs.

Figure 1 places the three on the request path. The response caches sit in an AI gateway or client library and end the request early on a hit. The prompt cache sits at the provider and changes only what the input costs.

Figure 1: A hit in either gateway cache ends the request before any provider is billed; a prompt-cache hit still runs the model and bills output.

How prompt caching works: the KV cache underneath

Prompt caching works by keeping the key-value (KV) cache a transformer computes for a prompt prefix and loading it for the next request that begins with the same tokens. The model skips recomputing attention over the cached prefix and processes only the new tokens, which is why the saving applies to input and not to output.

A KV cache is the set of attention key and value tensors a transformer builds for every token it has processed. During generation, each new token attends to all earlier ones, so the server keeps those tensors in GPU memory instead of recomputing them. The paper behind the vLLM serving engine describes this memory as “huge” and growing and shrinking with each request, and its PagedAttention design manages it in fixed-size blocks, reporting 2 to 4 times higher throughput than earlier systems at the same latency. The same paper notes that block-level management allows “sharing of KV cache within and across requests,” which is the mechanism prompt caching sells.

Open inference servers expose this directly. vLLM’s automatic prefix caching hashes each full block of tokens together with the hash of the block before it, so a block is identified by its entire prefix, and it evicts the least recently used blocks when memory runs short. SGLang uses a radix tree for the same purpose and reports up to 6.4 times higher throughput than other inference systems on its workloads. Teams that self-host models behind a router, such as the setups compared in the Frontier Wire survey of routers for vLLM, Ollama and cloud models, get prefix caching from the engine rather than from a price list.

Hosted providers turn the same idea into a billing line. OpenAI’s prompt caching guide states that caching “may store encrypted key/value tensors in GPU-local storage as application state,” and that cached states live on individual machines, which is why routing and traffic patterns affect the hit rate.

Prompt caching at Anthropic, OpenAI, Gemini and Bedrock

Every major provider now offers prompt caching, but they differ on whether it is automatic, how long entries live, the minimum prefix length and how writes are billed. Anthropic needs a cache marker or a top-level setting; OpenAI and Gemini cache implicitly by default; Bedrock supports both styles depending on the model.

A few details in the provider documentation change how a team should design prompts:

• Anthropic checks a prefix hash at each breakpoint and, on a miss, walks back at most 20 positions looking for an earlier match. Caches are isolated per workspace on the Claude API, and a hit needs “100% identical prompt segments, including all text and images.” If both cache_creation_input_tokens and cache_read_input_tokens are zero, the prompt was usually under the minimum length.
• OpenAI routes requests by a hash of the initial tokens and an optional prompt_cache_key, and warns that traffic above about 15 requests per minute for one prefix can overflow onto machines without the cache. Cached input tokens still count toward tokens-per-minute rate limits on OpenAI’s own API.
• Gemini offers no guaranteed saving on implicit caching. Its explicit context caching guarantees the discount but adds a storage charge billed on the TTL; Gemini 3.1 Pro Preview lists $4.50 per million tokens per hour on the pricing page. Explicit caching is not supported for Gemini 4 Argon and later, which rely on implicit caching.
• Bedrock states that cache hits on Claude models are not deducted against rate limits, and that cross-region inference “may lead to increased cache writes” at times of high demand.

The rate-limit difference matters for agents. A long-running coding agent that resends a large context every turn can be throttled by input-token quotas on one platform and not on another, at the same cache hit rate.

Prompt caching pricing and the break-even point

Prompt caching pays for itself after very few reuses because reads are discounted far more than writes are marked up. Anthropic’s pricing page states that a 5-minute cache “pays off after one cache read,” and a 1-hour cache after two. The saving is capped by the share of the bill that is cacheable input.

Using Anthropic’s list prices for Claude Sonnet 5.5, base input is $2 per million tokens, a 5-minute cache write is $2.50, a 1-hour write is $4, a cache read is $0.10 and output is $10. A 20,000-token prefix therefore costs $0.04 to send uncached, $0.05 to write into a 5-minute cache and $0.002 to read back. One read recovers the extra $0.01 of the write many times over. Anthropic adds that these multipliers stack with the Batch API’s 50 percent discount and with data-residency pricing.

Output is where the ceiling sits. Output tokens on the same model cost five times the input rate, and no provider discounts them through prompt caching. On a workload where output is a large share of spend, such as long generated reports or reasoning models whose hidden thinking is billed as output, prompt caching moves a smaller fraction of the bill. The Frontier Wire’s analysis of the cost problem in reasoning models covers why that share is growing.

The structure of the discount also favours certain workloads:

• Long, stable context, short questions. Document Q&A, support assistants with large policy documents and code assistants that resend a repository map get the largest relative saving.
• Multi-turn agents. Each turn resends the growing conversation, so a cache on the stable early turns pays back on every later turn. Anthropic’s automatic mode moves the breakpoint forward as the conversation grows for this reason.
• Bursty traffic for one prefix. Short TTLs reward traffic that keeps the cache warm. A prompt used once every 20 minutes misses a 5-minute cache every time and should use the 1-hour option or an explicit cache.

What breaks a prompt cache

A prompt cache breaks whenever any token before the breakpoint differs from the stored prefix. Anthropic and Bedrock evaluate the prefix in a fixed order, tools then system then messages, so an edit early in that order invalidates everything after it, even content that did not change.

Figure 2: Stable content goes first and the breakpoint goes after it; a change to an early block invalidates every block behind it.

Anthropic’s documentation lists the specific triggers. Changing tool definitions invalidates tools, system and messages. Toggling web search, citations or the speed setting keeps tools cached but invalidates system and messages. Changing tool_choice or adding or removing an image anywhere invalidates the messages level, and changing thinking parameters or effort settings always invalidates messages.

Most production misses come from application code rather than these documented cases. The common patterns are:

• Volatile values in the system prompt. A timestamp, request ID or user name near the top makes every prefix unique. Anthropic’s guidance is to place the breakpoint on the last block that is identical across requests, “not on a block that changes each time (such as a timestamp).”
• Non-deterministic serialization. Tool lists or JSON documents assembled from unordered maps can change order between requests with no change in meaning.
• Retrieval results placed early. A retrieval-augmented prompt that puts fresh search results before the instructions caches nothing useful. The trade-offs between retrieval and long context are covered in the Frontier Wire’s piece on RAG versus long context.
• Model switches mid-session. Caches are model-specific, so routing one conversation across several models, or falling back to another provider, starts a fresh cache each time.
• Clients that send no markers. On Anthropic models, a client that never sets cache_control caches nothing. Some agentic clients do exactly this, which is the problem gateway-side marker injection, discussed below, addresses.

Every provider reports cache reads and writes in the usage object (cache_read_input_tokens, cached_tokens, cacheReadInputTokens), and the ratio of reads to writes per prefix is the single most useful number for diagnosing a low hit rate.

How semantic caching works

Semantic caching stores each response with an embedding of the request that produced it and, for a new request, searches for a stored request whose embedding is close enough. If the similarity score clears a threshold, the stored response is returned and no model is called. The design needs an embedding model, a vector store, a threshold and an expiry policy.

The pipeline has four steps:

1. Embed. The incoming request, usually the latest user message and sometimes recent context, is converted to a vector by an embedding model. With OpenAI’s text-embedding-3-small at $0.02 per million tokens, embedding a 200-token question costs $0.000004.
2. Search. The vector is compared with stored vectors in a vector database such as Redis, Qdrant, Weaviate, Pinecone or Postgres with pgvector, typically by cosine similarity.
3. Decide. If the best match scores above the threshold, the stored response is served. Otherwise the request goes to the model.
4. Store. On a miss, the new response and its embedding are written with a time-to-live, ideally asynchronously so the caller does not wait.

GPTCache, an open-source library described in a 2023 workshop paper, popularized this pattern and reported responses 2 to 10 times faster on cache hits against OpenAI’s API. A 2024 study, GPT Semantic Cache, stored query embeddings in Redis and reported API calls reduced by up to 68.8 percent, with hit rates of 61.6 to 68.8 percent across its query categories and positive hit rates above 97 percent. Those figures come from the authors’ query sets; production hit rates depend on how repetitive real traffic is.

The costs run the other way on a miss. Every request that misses pays for the embedding call and the vector search on top of the full model call, so semantic caching adds latency and a small cost to traffic that rarely repeats. Further reading on the design choices includes a technical deep dive on semantic caching and a guide to cutting cost and latency with semantic caching at scale.

The threshold problem in semantic caching

A semantic cache is only as safe as its similarity threshold, and one fixed threshold cannot be right for every request. Two questions can score as near-identical yet need different answers, such as the same question about two different products, while true paraphrases can score below the cut-off and miss.

The vCache paper, accepted to ICLR 2026, states the problem directly: “static thresholds do not give formal correctness guarantees,” which leads to unexpected error rates and lower hit rates. Its authors propose learning a separate threshold for each cached prompt online, with a user-specified maximum error rate, and report up to 12.5 times higher hit rates and up to 26 times lower error rates than static-threshold and fine-tuned-embedding baselines on their benchmarks.

A post on the Bifrost blog, on the grey-zone problem in semantic caches, examines the band of scores just below the threshold where many requests share intent with a cached answer. It describes resolving those cases after the response is served, with a judge model that approves or rejects promotion of a cached answer, so serving latency is unchanged.

Until such methods are standard, operators manage the risk with scoping rather than a single number:

• Partition the cache. Keep entries per tenant, per feature or per user, so a similar question from a different customer never receives another customer’s answer.
• Exclude what cannot be replayed. Personalized answers, tool calls, and anything that depends on current data or account state should bypass the cache or carry a short TTL.
• Raise the threshold where errors are costly. A support FAQ can tolerate a looser match than a billing or medical answer.
• Limit conversation depth. Matching on a single question is safer than matching mid-conversation, where the same words can mean different things.
• Log similarity scores. Recording the score on every hit lets a team audit borderline matches and tune the threshold from evidence.

Exact-match caching: the cheap first layer

Exact-match response caching hashes the normalized request and replays the stored response only when a later request is identical. It needs no embedding model and cannot return an answer to a different question, which makes it the safe default for deterministic, repeated calls such as classification, extraction or evaluation runs.

Several AI gateways offer exact matching alone or as the first stage before semantic search:

• Cloudflare AI Gateway caches on an exact match of the entire request, keyed on a SHA-256 hash of provider, endpoint, model, auth header and body. TTLs run from 60 seconds to one month, and the documentation says semantic search for caching is planned for the future.
• LiteLLM supports exact caching in memory, on disk, in Redis or Valkey and in S3 or GCS buckets, plus semantic caches on Redis, Valkey or Qdrant.
• Kong’s AI Semantic Cache plugin combines exact and semantic caching on Redis or Postgres with pgvector, respects standard Cache-Control directives and reports hits in an X-Cache-Status header. Kong lists it as part of its AI Gateway Enterprise offering.

Exact matching also does more for agent and batch workloads than its name suggests. Evaluation suites, retried requests and scheduled jobs often send byte-identical requests, and a hash lookup in Redis adds well under the time of a provider call. A separate survey of AI gateways for semantic caching compares how gateways layer the two match types.

A worked cost example

Prompt caching and semantic caching stack because they remove different costs. In this example, prompt caching removes most of the input bill and leaves output untouched; a semantic cache then removes whole calls, so most of what it saves is output. The figures use Anthropic’s list prices for Claude Sonnet 5.5 and OpenAI’s embedding price.

The workload is a support assistant with a 20,000-token system prompt of instructions and product documentation. Each request adds a 200-token question and receives a 400-token answer, at 10,000 requests a day with steady traffic, so the 5-minute cache stays warm and cache writes are rare enough to ignore.

The semantic-cache rows include the embedding of each 200-token question, about 2 million tokens a day at $0.02 per million, or $0.04. The 20 percent hit rate is an assumption for illustration, well below the rates reported in the GPT Semantic Cache study; actual rates must be measured.

Three conclusions follow from the table:

• Prompt caching is the larger lever when the prefix dominates. It removes about $380 a day on its own, more than four times what the semantic cache alone saves.
• After prompt caching, output is most of the bill. In the prompt-cached setup, $40 of the $64 daily cost is output, which only a response cache can remove.
• What the semantic cache embeds matters. Embedding the full 20,000-token prompt instead of the question would cost about $4 a day and make every request look similar to every other, because the shared system prompt would dominate the vector. Caches that let operators exclude the system prompt from the match avoid both problems.

The arithmetic matches the approach in the Frontier Wire’s guide to LLM cost management at the gateway, which treats caching as one control alongside budgets and routing. An article on reducing OpenAI and Anthropic bills with semantic caching works through similar figures for chat workloads.

How Bifrost runs both caches at the gateway

Bifrost is used here as the worked example because its documentation covers both cache types and how they interact. Bifrost is an open-source AI gateway written in Go and released by Maxim AI under the Apache 2.0 license. It exposes 25+ providers and 10,000+ models through one OpenAI-compatible API, and its published benchmark reports 11 µs of added overhead per request at 5,000 RPS on an AWS t3.xlarge.

Its semantic caching has two lookup paths. Direct matching hashes the normalized request, its parameters and the stream flag, and needs no embedding provider. Semantic matching embeds the request and serves a stored response above a cosine similarity threshold, 0.8 by default. When both are on, the direct lookup runs first and the semantic search runs only on a direct miss. Entries expire after a default TTL of five minutes and are written asynchronously, so a miss does not wait on the cache write.

Figure 3: Response caching and prompt-cache marker injection are separate, opt-in settings, and one request can use both.

The documented defaults line up with the scoping practices described above:

• Opt-in per request. Caching engages only when a request carries a cache key in the x-bf-cache-key header or the operator sets a default key, so no caller receives a replayed answer by accident.
• Identity scoping. Entries are scoped by default to the virtual key and user that wrote them, so two identities never share an entry even with the same key and prompt.
• Tool calls excluded. Responses that carry tool calls are not cached by default, because a replayed tool call would run in another caller’s agent loop.
• Conversation limit. Caching is skipped when a conversation has more than three messages by default, and the system prompt can be excluded from key generation.
• Stricter-only overrides. A per-request threshold header can raise the threshold but never lower it below the configured value.

The cache needs a vector store for both modes, with Redis or Valkey, Weaviate, Qdrant and Pinecone supported, and it covers chat and text completions, the Responses API, embeddings, transcription, speech and image generation, including streaming. Every lookup returns x-bifrost-cache-* response headers with the hit type and, on a semantic hit, the similarity score. The model catalog prices a direct hit at zero and a semantic hit at the embedding call only, and on a miss it records the model cost plus the embedding.

On the prompt-caching side, Bifrost’s auto prompt caching setting addresses clients that send no cache markers. The documentation names Codex as an example of an agentic client that sends none, which on Anthropic models means nothing is cached. With auto_inject turned on for a provider, the gateway marks the first cacheable block of requests that arrive without markers, so the first turn writes the cache and later turns read it. The setting is off by default, caller markers always take precedence, at most four markers are added, and the marker is translated per provider: cache_control for Claude on Anthropic and Vertex AI, cachePoint on Bedrock and prompt_cache_breakpoint for OpenAI’s GPT-5.6 family. On Gemini and other implicit-caching providers it does nothing.

The Bifrost team’s post on LLM cost optimization at the gateway describes how these settings combine with routing and reasoning budgets, and a companion piece on semantic caching and dynamic routing covers the same pairing. Agent clients such as Claude Code and Codex, and the gateways that sit in front of them, are compared in the Frontier Wire’s survey of gateways for coding agents.

Choosing a caching strategy by workload

The right mix depends on what repeats. Repeated context calls for prompt caching; repeated questions call for a response cache; identical requests call for exact matching first. Most production systems end up with prompt caching everywhere it applies and a scoped response cache on the few endpoints with genuinely repetitive traffic.

A practical rollout order follows from that table:

1. Restructure prompts for prefix stability. Move static instructions, tools and documents to the front, remove volatile values, and make serialization deterministic.
2. Turn on provider prompt caching and confirm reads in the usage fields. For clients that send no markers, add them in the client or at a gateway.
3. Add exact-match caching for deterministic, repeated calls.
4. Add semantic caching last, on specific endpoints, with a cache key that partitions by tenant or feature, a conservative threshold, a short TTL, and similarity logging from day one.

Routing decisions interact with all of this. A failover to a second provider starts a cold prompt cache there, which is worth factoring into failover and load-balancing policies that move traffic often.

What to cache, and where

Prompt caching and semantic caching answer different questions about the same bill. Prompt caching asks whether the model has to reread context it has already processed, and the answer from every major provider is now no, at 5 to 10 percent of the input price on current models. Semantic caching asks whether the model has to be called at all, and the answer depends on how repetitive the traffic is and how much error the application can tolerate.

For most teams, prompt caching is the first and safest saving: it never changes an answer, and the work is largely prompt hygiene. A semantic cache is a narrower tool that pays off on repetitive, low-risk endpoints and needs scoping, thresholds and monitoring to stay correct. Placing the response caches at a gateway, as covered in the Frontier Wire’s guide to gateway-level LLM cost management, keeps those controls in one place rather than in every service. What would change this picture is wider adoption of verified semantic caches with per-prompt error bounds, which would make the response cache safe to apply far more broadly than a single similarity threshold allows today.