A model has never seen your documents, and there are two ways to tell it about them. You can send the text along with the question, or you can build a search step that pulls out only the relevant fragment. Both work. They cost different amounts and break in different ways.
How the two approaches differ
Long context means a bigger prompt. You take your documents, concatenate them and send everything to the model next to the question. It decides for itself which parts matter.
RAG, retrieval-augmented generation, is built another way. The text is cut into chunks ahead of time, and each chunk becomes an embedding, a vector that encodes meaning. When a question arrives, the system looks for the chunks closest to it in meaning, and only that selection goes into the prompt. The corpus stays outside.
On paper the gap looks small. The invoice and the answer quality say otherwise.
What it does to the bill
With long context, every call pays for the entire corpus as input tokens. A question about one paragraph out of a thousand pages costs the same as a question about all of them. The corpus grows, and the price of each request grows with it, linearly and with no ceiling.
Prompt caching helps a little. If the same large context is reused again and again inside the cache window, you pay full price once and input is cheaper after that. When questions vary and the corpus keeps changing, caching does not save you.
With RAG, the price of a request depends on the size of the slice you hand the model, not on the size of the corpus. The knowledge base can reach a million documents and the prompt still gets a handful of chunks. You pay elsewhere: a vector store, an embeddings pipeline and, sometimes, one more model call to rerank the candidates.
What it does to accuracy
The weak spot of long context is well known: a model holds details from the middle of a large text less reliably than details from the start or the end. The longer the context, the better the odds that the fact you need drowns in noise.
RAG fails differently. Accuracy is capped by the quality of search. If chunking is coarse and the right fragment misses the retrieved set, the model answers with confidence and gets it wrong, because it never saw the data. The failure is quiet. No log line, no apology, just a wrong answer.
What you have to maintain
Long context is simple by construction: there is no separate pipeline, so nothing breaks overnight. The trade is that the token budget is yours to watch, and if the corpus keeps growing you will eventually hit the model's window.
RAG is a whole system. You choose a chunking strategy, pin the version of the embeddings model, rebuild the index on every content update and check that search keeps finding what it should. Any one of those parts can decay without a sound. Usually you hear about it from users complaining about odd answers, not from an alert.
The hybrid most teams settle on
Few teams pick one approach in its pure form. A small, stable corpus, such as one project's docs or a team knowledge base, usually goes into context whole, especially when caching is available.
A growing or mixed corpus usually moves to RAG with one adjustment: search returns whole relevant documents instead of short snippets, and those documents then go into context in full. You get the precision of search plus the model's ability to reason over complete text, not over a sentence torn out of its page.
An example. A support bot built on five help articles lives happily on long context, since all five fit in the prompt with room to spare. The same bot over three hundred pages of API documentation, where the authentication section changes every week, usually moves to search. The whole corpus no longer fits in context without the model losing track of details.
How to decide for your case
Three questions settle most cases.
First: how big is the corpus and how fast does it grow? If it fits in context with room to spare and rarely changes, you do not need search.
Second: how often does the data change? If content updates every day, the index has to be rebuilt every day, and that costs money and team time.
Third: do you need one exact fact or a synthesis across many sources? An exact fact is better found through search and quoted. A synthesis over a large body of material is often easier on full context.
Putting a number on the difference is easier than it sounds. The codexkey calculator on the site estimates the cost of a request at different context sizes, and you can compare that with the pair of calls RAG usually needs, reranking included.
In short
Long context is simpler and, at moderate volume, more accurate, but the bill grows with the corpus. RAG keeps the bill flat at any volume, at the price of extra infrastructure and the risk of missing the right chunk. Small, stable corpus: context. Growing and mixed: search that gives the model whole documents, not fragments.