RAG vs. long context: choose by failure mode, not by benchmark

TL;DR

• Long context fails quietly and late: models still lose accuracy as prompts grow, even when they pass needle-in-a-haystack tests, and the miss is hidden inside a fluent answer.
• Retrieval fails loudly and early: if the retriever does not return the right passage the model cannot recover, but the failure is visible and measurable before generation.
• As of 27 September 2026, Google and OpenAI charge double for input above 200k and 272k tokens respectively; Anthropic does not. Prompt caching cuts repeated long-context reads to 10 percent or less of base price.
• The published head-to-head study found long context more accurate and retrieval far cheaper, and that a router between them kept the accuracy at a fraction of the cost.

Every frontier API now accepts roughly a million tokens, so asking whether to build retrieval infrastructure at all is reasonable. This piece compares the two approaches on stated criteria, using the papers that measured them and the vendor pricing pages as they read today. The choice should be made by asking which kind of failure you can afford, because neither approach eliminates failure; it moves it.

Two ways to put documents in front of a model

Retrieval-augmented generation (RAG) means storing documents outside the model, searching them per query, and pasting the top results into the prompt. The term comes from Lewis et al. (2020), who paired a generator with “a dense vector index of Wikipedia, accessed with a pre-trained neural retriever” and reported that the result could “generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline”.

Long context means skipping the search and sending the documents whole. The windows that make this plausible are now standard. Anthropic’s context windows documentation gives Claude Opus 5.5, Sonnet 5 and Fable 5.1 a 1M-token window by default, which it says is “roughly 555k words” on the current tokenizer. Google’s model page for Gemini 3.1 Pro Preview lists an input limit of 1,048,576 tokens. OpenAI’s page for GPT-6 Sol lists a “1,050,000 context window” with a maximum of 922,000 input tokens.

Google’s own long context guide says the window “invites a more direct approach: providing all relevant information upfront”. The same page concedes the limit: single-fact retrieval reaches “up to 99% accuracy in many cases”, but with multiple facts to find “performance can vary to a wide degree”. That gap between one needle and many is where the research lives.

How long context fails

The foundational result is Liu et al. (2023), “Lost in the Middle”. The authors gave six models, including GPT-3.5-Turbo and Claude 1.3, a question plus 10, 20 or 30 documents, and moved the one containing the answer through the context. They found “a distinctive U-shaped performance curve”: accuracy is highest with the answer at the start or the end, and “significantly degrades when models must access and use information in the middle”. For GPT-3.5-Turbo the drop was “more than 20%”, and in the worst case accuracy fell below the 56.1 percent the model scored with no documents at all. Extended-context variants were “not necessarily better at using their input context”.

Those models are retired, but later benchmarks show the shape persists. RULER (Hsieh et al., 2024) tested 17 models on 13 tasks and reported that “despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases”; although all claimed 32K or more, “only half of them can maintain satisfactory performance at the length of 32K”. NIAH, needle in a haystack, means finding one planted sentence in filler text.

NoLiMa (Modarressi et al., 2025) explains why passing NIAH is not enough. Its needles share almost no words with the question, so the model must infer the link rather than pattern-match. Across 13 models that claimed at least 128K, “at 32K, for instance, 11 models drop below 50% of their strong short-length baselines”. GPT-4o fell “from an almost-perfect baseline of 99.3% to 69.7%”, and “even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts”.

LongBench v2 (Bai et al., 2024) moves from synthetic needles to 503 realistic multiple-choice questions over contexts of 8k to 2M words. Human experts scored 53.7 percent under a 15-minute limit. The best model answering directly scored 50.1 percent; o1-preview, with longer reasoning, reached 57.7 percent. Extra inference compute helps here, a theme covered in our test-time compute explainer. OpenAI’s own eval, OpenAI-MRCR, plants two, four or eight identical requests in a conversation and asks for a specific one, with bins up to 1,048,576 tokens.

Vendors now acknowledge the pattern. Anthropic’s context windows page states: “As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”

The failures that are not wrong answers

Databricks’ Long Context RAG Performance of LLMs (August 2024) fed 13 models retrieved passages at lengths up to 192k tokens and catalogued how they broke. GPT-4 mostly gave wrong answers. Claude 3 Sonnet mostly refused, citing copyright, rising from 3.7 percent of responses at 16k to 49.5 percent at 64k. Mixtral repeated itself; DBRX summarised instead of answering. GPT-4o and Claude 3.5 Sonnet showed “little to no performance deterioration” in the range tested. The authors blamed a “lack of sufficient long context post-training”. Long-context failure is not one thing, and none of the variants announces itself.

How retrieval fails

Retrieval fails at the retriever. If the right passage is not in the returned set, no model quality can compensate, so a mediocre retriever caps the whole system. The most careful account of when this happens is Li et al. (2024), the Google study that compared RAG with long context (LC) on GPT-4o, GPT-3.5-Turbo and Gemini 1.5 Pro across nine datasets. Its headline: “when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG’s significantly lower cost remains a distinct advantage.” On Gemini 1.5 Pro with the Dragon retriever, LC averaged 49.70 against RAG’s 38.09.

The authors classified RAG’s misses into four kinds: queries that need “multi-step reasoning so the results of previous steps are needed to retrieve information for later steps”; general queries such as “What does the group think about XXX”; long and complex queries; and implicit queries that demand “a thorough understanding of the entire context”. Each is a case where the answer is in the corpus but the question does not resemble the passage: the same lexical-mismatch weakness NoLiMa found in long-context attention, one stage earlier in the pipeline.

Long context hides its misses inside a fluent answer. Retrieval shows you its misses before the model speaks.

The redeeming property is that the retrieved set is a concrete artefact. You can log it, score its recall against known source passages, and fix chunking or embeddings when recall is low. A long-context call gives you nothing equivalent to inspect.

The criteria

The table states the criteria before the verdict, so the weighting can be argued with.

What each costs in September 2026

Prices below were read from the vendors’ pages on 27 September 2026, per million tokens. They change often; the pattern matters more than the digits.

Sources: OpenAI pricing and the GPT-6 Sol model page, the Gemini API pricing page (last updated 24 September 2026), and Anthropic’s pricing documentation.

The arithmetic follows from the list prices. A 300,000-token corpus sent whole to Claude Sonnet 5 costs $0.60 in input per query; on GPT-6 Sol the same request crosses 272K and costs $1.20; on Gemini 3.1 Pro Preview it crosses 200k and also costs $1.20. A retrieval pipeline that sends 8,000 tokens of passages to Sonnet 5 costs about $0.016 per query, roughly 37 times less, before the cost of running the index. At a thousand queries a day that is $600 against $16. On the tiered providers output is dearer too, at 1.5 times above the threshold.

Prompt caching as the cost lever

All three providers cache repeated prompt prefixes, and this is what makes long context financially defensible for a stable corpus.

Anthropic’s prompt caching documentation prices a five-minute cache write at 1.25 times base input, a one-hour write at 2 times, and a cache read at “0.1 times the base input tokens price”, with exceptions of 0.05 times on Opus 5.5 and 0.025 times on Fable 5.1. The minimum cacheable prompt is 1,024 tokens on Sonnet 5. A read also refreshes the cache, so steady traffic keeps it warm.

OpenAI’s prompt caching guide says caching is automatic on shared prefixes of at least 1,024 tokens for GPT-5.6 and later, that a cached prefix “remains eligible for reuse for 30 minutes after its most recent write or reuse”, and that the read charge is “0.1× the uncached input-token rate”. Google’s context caching page says “implicit caching is enabled by default for all Gemini 2.5 and newer models”, with a 4,096-token minimum on Gemini 3.x; the pricing page adds an hourly storage charge the other two do not levy.

With a warm cache the 300,000-token corpus on Sonnet 5 drops from $0.60 to $0.06 per query. That is still nearly four times the retrieval pipeline’s $0.016, but it is the same order of magnitude, and it buys the accuracy advantage Li et al. measured. Caching does nothing for a corpus that changes between requests, for multi-tenant systems where every customer’s documents differ, or for traffic too sparse to keep the prefix alive. Those are the conditions under which retrieval was already the right answer.

A decision rule

Use long context when the corpus fits in a few hundred thousand tokens, changes rarely, the questions are the general or multi-step kind that defeat retrievers, and the traffic is steady enough to keep a cache warm. A single contract, one codebase module, a regulatory filing.

Use retrieval when the corpus is larger than a window, is updated continuously, differs per tenant, or when per-query cost is the binding constraint. A support knowledge base, a ticket history, a document store shared across customers.

Use a router when you can afford neither the cost of always sending everything nor the misses of always retrieving. Li et al.’s Self-Route asks the model whether the retrieved passages suffice and escalates to the full context only when they do not. RAG and LC gave “exactly identical” predictions on 63 percent of queries, and the router cut cost “by 65% for Gemini-1.5-Pro and 39% for GPT-4O” while matching long-context accuracy. An AI gateway, which already sits between the application and every model call, is a natural place for that logic; see our field notes on running one.

What to measure

Whichever path you take, build a question set with known answers and known source passages, and score two things separately: whether the right passage reached the model, and whether the model answered correctly given that it did. Low recall with high conditional accuracy is a retriever problem. High recall with low accuracy is a model or prompt problem, and for long context it signals that you have entered the range where NoLiMa and RULER say accuracy falls. Test at the prompt lengths you will actually send, not at the advertised window; every study here found the two differ. Our comparison of LLM evaluation frameworks covers the tooling for keeping those scores over time.

The limits are worth stating. The head-to-head study dates from 2024 and used models two generations old; the current long-context benchmarks test comprehension, not the RAG-versus-LC question directly. No vendor publishes accuracy curves for its own models at full window on a NoLiMa-style task. Until one does, long context is more capable and more expensive, retrieval is cheaper and more brittle, and your own failure rate is the number to trust.