RAG with Claude in 2026: accurate retrieval, lower cost

Retrieval-augmented generation in 2026 looks little like what teams built two years ago. Claude Sonnet 4.6 has a 200K token context window, and prompt caching cuts the price of stable context by an order of magnitude. The question is no longer "how do I squeeze the right chunk into 8K tokens?" It is "do I need retrieval at all, or is it simpler to put the whole knowledge base into a cached system prompt?"

What follows is a working guide for the architect. When classic retrieval still wins. How to use long context without going broke. Which embedding models and rerankers to pair with Claude. How to get citations you can trust. And the pitfalls that show up in production again and again.

Classic RAG vs long context: when to retrieve

Start by asking whether you need retrieval. The 200K window on Sonnet 4.6 holds roughly 100-150 medium-sized documents (5-10 pages each), and for plenty of real knowledge bases that is the whole corpus. Three rules of thumb:

• Put everything in context if the corpus is small (under ~150K tokens), stays the same between sessions, and questions cut across several documents. A cached system prompt with the full corpus beats a retriever on quality and on latency, starting from the second call.
• Retrieve if the corpus is large (millions of tokens), changes often, or each query needs only 1-5 documents. Embedding search is still the cheapest way to narrow the field.
• Go hybrid: retrieve generously (top-50 or top-100) and let Claude's long context absorb the rest. In 2026 this is the dominant pattern, because it lowers the demands on retrieval precision and shifts more of the quality work onto the model.

Why prompt caching changes the math

Prompt caching is the main reason long context became affordable. You mark a prefix of the prompt as cacheable, Anthropic stores it on its side, and later calls with the same prefix pay roughly 10% of the normal input rate for that part. Writing to the cache costs slightly more than a normal input token. Reading from it is dramatically cheaper.

For a stable knowledge base this works out as follows:

1. First call: full price to write the cache (~25% premium on the cached part).
2. Every later call within the TTL: ~10% for the cached prefix, and full price only for the unique tail of the request.

In practice a RAG assistant that answers dozens of queries an hour against one corpus pays back the cache in two or three calls and then runs about 10× cheaper than it would without it. That is why "load 80 documents into Sonnet" is a real production pattern now and not a demo trick.

Embedding model choice

If you do need retrieval, the embedding model matters more than most people expect. The practical shortlist for 2026:

• Voyage AI (voyage-3-large, or voyage-code-3 for code): the pairing Anthropic officially recommends. Strong on English, code and multilingual text, and built specifically for RAG.
• OpenAI text-embedding-3-large: a safe fallback if you already have OpenAI keys and want a proven 3072-dimensional baseline.
• BAAI bge-m3: open-source, multilingual, strong recall. It runs locally on a single A100 when embeddings have to stay on-prem.

Whatever you choose, lock the model version. Re-embedding the corpus because a vendor quietly updated a model is the most common self-inflicted RAG outage in production.

Reranking: the cheap win everyone makes now

A vector search returns plausible candidates, not a finished ranking. The standard 2026 setup: over-retrieve (top-50 or top-100), then cut down to the top-10 or top-20 with a cross-encoder:

• Cohere Rerank 3: hosted, multilingual, very good quality, pay-per-call.
• BAAI bge-reranker-v2-m3: open-source, runs on commodity GPUs, no per-call fee.

A reranker usually lifts answer quality more than swapping Sonnet for Opus does, and it costs a fraction as much. If you still go without one, that is the first thing to fix.

Hybrid search: BM25 still earns its place

Pure vector search misses queries that need an exact match: product SKUs, error codes, function names, section numbers of a law. The cure is hybrid search. BM25 (via OpenSearch, Elastic or tantivy) and vector retrieval run in parallel, and the scores are merged with reciprocal rank fusion (RRF). On mixed corpora hybrid reliably beats vector-only, and if the documents are already indexed it costs almost nothing.

Chunking strategies that work

Chunking is still a tuning problem, not a solved one. The defaults that hold up in 2026:

• 800-1200 tokens per chunk with 100-200 tokens of overlap for ordinary prose.
• A whole function for code, with the file path and class name in the header.
• Semantic chunking (split on heading boundaries first, then by size) for technical documentation.
• Always store a parent document ID and a stable chunk ID, so a citation can be traced back to its source.

Do not cut too small. Chunks under 500 tokens lose context and make the reranker do work the embedder should have done.

Reliable citations

Invented citations are the fastest way to burn user trust. The pattern that works on Claude:

1. Tag every retrieved chunk in the prompt with an explicit ID: [1] (source: docs/auth.md): ...content...
2. In the system prompt, require Claude to tag every factual claim with a bracketed ID and to decline to answer when no retrieved chunk supports it.
3. Post-process the response and check that every [N] maps to a chunk you actually sent. Drop or flag the ones that do not.

Step 3 is not up for debate. Even Sonnet 4.6 sometimes makes up a citation ID under pressure, and the verifier is what makes the system deserve trust.

A minimal pipeline on codexkey

Here is a working sketch of the full loop (embed, retrieve, rerank, cite) through the codexkey endpoint:

import anthropic
import voyageai
import cohere

client = anthropic.Anthropic(
    base_url="https://api.codexkey.online/v1",
    api_key="<your-codexkey-key>",
)
voyage = voyageai.Client()
co = cohere.Client()

def answer(question: str, corpus_chunks: list[dict]) -> str:
    # 1. Embed the query
    q_emb = voyage.embed([question], model="voyage-3-large").embeddings[0]

    # 2. Vector retrieve top-50 (your vector store call here)
    candidates = vector_store.search(q_emb, k=50)

    # 3. Rerank to top-10 with Cohere
    reranked = co.rerank(
        model="rerank-3",
        query=question,
        documents=[c["text"] for c in candidates],
        top_n=10,
    )
    top = [candidates[r.index] for r in reranked.results]

    # 4. Build the cited context
    context = "\n\n".join(
        f"[{i+1}] (source: {c['source']}): {c['text']}"
        for i, c in enumerate(top)
    )

    # 5. Call Claude with a cached system prompt
    msg = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": "You are a precise assistant. Cite every factual "
                        "claim with the bracketed source ID like [1]. "
                        "If no source supports an answer, say so.",
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=[
            {
                "role": "user",
                "content": f"{context}\n\nQuestion: {question}",
            }
        ],
    )
    return msg.content[0].text

The cache_control marker switches on the 10× discount on the system prompt across calls. If the knowledge base is stuffed in whole, move the corpus itself into the cached system block instead of the user message. Same pattern, noticeably bigger savings.

Evaluate retrieval and generation separately

The most useful habit in production RAG is measuring the two stages independently.

• Retrieval quality: recall@k against a labelled set of (question, correct document) pairs. If recall@10 is below ~0.85, no amount of prompt engineering will rescue the answer.
• Generation quality: faithfulness (does the answer agree with the retrieved context?) and answer correctness. People score it, or an LLM judge does on a held-out eval set.

When the metrics are mixed, teams end up "fixing" generation problems by tuning the retriever and the other way round. Keep them apart.

Common pitfalls

• Lost-in-the-middle. Even at 200K tokens, Claude pays the most attention to the start and end of the context. Put the most relevant chunks first, the question last, and do not bury critical facts in the middle of a 100K token wall.
• Citation hallucination. Always check that cited IDs point to real chunks. An unverified citation is a failure, not a warning.
• Cache invalidation. Any edit to the cached prefix, even a single space, resets the cache. Version your system prompts and templates.
• Stale embeddings. When you change the embedding model, re-embed the whole corpus. A vector store holding a mix of models degrades silently.
• Over-retrieval without reranking. Taking the top-100 chunks and sending them straight to the prompt is expensive and worse in quality. Always rerank.

Bottom line

The winning 2026 RAG stack is boring in the best sense: a cached system prompt, Sonnet 4.6 as the generator, Voyage embeddings, a Cohere or BGE reranker, hybrid BM25 plus vector retrieval and a citation verifier on the way out. Each part is cheap, well understood and improves quality on its own. Together they make a system that is fast, auditable and steady in price while the corpus grows. For prices on Sonnet 4.6 and the rest of the family, see the companion post on Claude API pricing in 2026.