Claude 200K context: when to stuff and when to retrieve

Claude's 200K-token context window sounds infinite until you do the math. 200K tokens is about 150,000 words, roughly 600 paperback pages, or a mid-sized codebase with its tests. So why are we still building RAG pipelines in 2026? Because "fits in the window" and "belongs in the window" are different questions. Mixing them up has become the most expensive mistake teams make with Claude this year.

What follows is the decision framework we use ourselves. It has three working strategies: full stuff, RAG with top-N stuffing, and a hybrid with re-rank. For each, it shows when it wins on cost, latency and answer quality.

How much fits in the window

A 200K window holds roughly:

• ~150,000 English words (the average token is about 0.75 of a word in ordinary English).
• ~500-800 pages of typical prose, depending on density.
• ~30,000-50,000 lines of code in mainstream languages (TypeScript, Python, Go).
• The backend of a medium SaaS product, the full database schema and three weeks of tickets, with room to spare.

The urge to paste everything in is strong, and sometimes it is right. But two forces push against it: input pricing and lost-in-the-middle.

Force 1: at scale, input pricing outweighs everything

In long-context calls, output is rarely what drives cost. A request with 180K tokens in and 1K out has 180 times more input tokens than output tokens, so you are really paying for the input. At volume, the question of cost comes down entirely to input price.

That is why prompt caching changes the rules (more below). Without a cache, 180K tokens in every request is a tax on every call. That tax is what sinks naive long-context strategies financially at around 1,000 requests a day.

Force 2: lost-in-the-middle is real, even on Claude

For several generations in a row, Claude has been the best long-context model on needle-in-a-haystack benchmarks. It really is better than most competitors at pulling a fact from an arbitrary spot. But "better" is not "perfect".

In our internal evals at depths from 150K tokens, the picture looks like this:

• Start of context (first 10K tokens): retrieval accuracy ~99%.
• End of context (last 10K tokens): accuracy ~98%.
• Middle (80K-120K depth): ~92-95% on multi-hop reasoning, lower on subtle facts.

A gap of 4-7 points matters. If your application has to find one clause buried 100K tokens deep every time, "usually works" is not something you can ship.

Three strategies

Strategy 1: full stuff with prompt caching

Use it when: the context is large but stable across calls. Documentation, fixed knowledge bases, a frozen repository under analysis, customer manuals people ask about all the time.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  baseURL: "https://api.codexkey.online/v1",
  apiKey: process.env.CODEXKEY_API_KEY,
});

const KNOWLEDGE_BASE = await loadFullDocs(); // ~180K tokens, stable

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: KNOWLEDGE_BASE,
      cache_control: { type: "ephemeral" },
    },
  ],
  messages: [{ role: "user", content: userQuestion }],
});

The first call writes the cache (1.25x the input price). Every later call within the TTL reads it at 0.1x, which is 12.5 times cheaper. Two reuses already beat the no-cache baseline. After ten you are paying roughly 10% of what you would have paid otherwise.

Strategy 2: RAG with top-N stuffing

Use it when: the corpus is well beyond 200K (knowledge graphs, archives of millions of documents, years of ticket history) and each query touches its own slice.

const topChunks = await vectorSearch(userQuery, { k: 20 });
const reranked = await rerank(userQuery, topChunks, { keep: 8 });

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: `Context:\n${reranked.map(c => c.text).join("\n---\n")}\n\nQuestion: ${userQuery}`,
    },
  ],
});

Input stays small (5K-20K tokens), latency is low and cost is predictable. The compromise lies elsewhere: retrieval quality becomes the bottleneck. A weak embedding model or poor chunking ruins answers, however strong Claude is.

Strategy 3: hybrid with re-rank

Use it when: answer quality matters more than cost and latency. Legal work, medicine, high-value support, code review on large repositories.

const stableContext = await loadFrameworkDocs(); // 80K tokens, cached
const dynamic = await vectorSearch(userQuery, { k: 30 });
const reranked = await rerank(userQuery, dynamic, { keep: 12 });

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 2048,
  system: [
    { type: "text", text: stableContext, cache_control: { type: "ephemeral" } },
  ],
  messages: [
    {
      role: "user",
      content: `Relevant excerpts:\n${reranked.map(c => c.text).join("\n---\n")}\n\nQuestion: ${userQuery}`,
    },
  ],
});

Stable framework and policy docs live in a cached system prompt. Volatile, per-query material is retrieved, re-ranked and added fresh each time. You get the recall of RAG, the depth of a stuffed context and most of the savings from caching.

The decision matrix

One pattern covers all of it: cache the stable part, retrieve the volatile part.

How to measure

You cannot choose a strategy without measurements. Here is our standard test set for long-context decisions:

1. Needle-in-a-haystack at varying depths. Insert a unique fact at 10%, 50% and 90% of the context, 50 queries per depth. Measure retrieval accuracy.
2. Multi-hop reasoning. Put two related facts at different depths and ask a question that needs both. Degradation here is steeper than with a single needle, especially in the middle band.
3. Cost per resolved query. Divide total spend by the number of correctly answered test cases. This is the only metric that matters for a budget.
4. P95 latency. Long context means a slow first token. RAG with 8K of input typically gives a time to first token 3-5 times faster than a fully stuffed 180K.
5. Cache hit rate. If caching runs in production, instrument it. A hit rate below 70% means your TTL or partitioning is wrong.

How prompt caching changes the arithmetic

Without caching, 180K tokens stuffed in 100 times cost 100x the input price of a single call. With caching:

• Call 1: 1.25x (cache write).
• Calls 2-100: 0.1x each (cache reads).
• Total: 1.25 + 9.9 = 11.15x for 100 calls.

That is roughly 9 times cheaper than the same workload without a cache. Break-even arrives at two reuses, and after that the gap only widens.

This one feature is what makes "paste in all the docs" viable in 2026, when in 2024 it was financial nonsense. If your context is stable and you are not caching, you are leaving money on the floor.

Bottom line

The 2026 long-context playbook boils down to one rule: cache the stable part, retrieve the volatile part.

• Small, stable context: stuff it in whole and cache it.
• Huge context that differs per query: use RAG.
• Both at once (the realistic case for most production apps): split them. Put the stable part in a cached system prompt, then retrieve and re-rank the volatile part into messages.

Choose a strategy by measurement, not by feel. Run the needle-in-a-haystack eval, track cost per resolved query, watch the cache hit rate. The teams getting the most out of Claude this year are not the ones with the cleverest prompts. They are the ones who measured before they shipped.