Do We Still Need RAG?

There is a fundamental truth about large language models (LLMs): they are "frozen in time." They know everything about our world up to the date their training ended — and absolutely nothing about what happened five minutes ago. Likewise, they know nothing about your private data, internal wikis, or proprietary codebases. And if we want a model to have access to any of this information, we need to solve the problem of context injection: how do we deliver the right data to the model at the right moment?

Today, there are two fundamentally different approaches to solving this problem.

Retrieval-Augmented Generation

The first is, roughly speaking, the engineering path: RAG, or Retrieval-Augmented Generation. The pattern looks like this: we have an LLM and an incoming user query. In advance, we take the documents we want to provide to the model — PDFs, source code, or entire books — split them into small fragments (chunks), and pass them through an embedding model. That model converts the fragments into vectors, which are then stored in a specialized vector database.

When a user asks a question, the system performs semantic search, retrieves the most relevant fragments, and embeds them in the model's context window. Thus, the context window contains not only the user query, but also all fragments retrieved from the vector database — together they form the final context.

This approach works, but it relies on one important assumption: that your retrieval logic actually found the right information in the vector database.

Long Context

The second approach is more "brute force," but native to the model itself. It is called "long context" (Long Context). Here we bypass databases and embedding models entirely: we simply take documents and load them directly into the context window, letting the model's attention mechanism do the heavy lifting of finding the answer.

For a long time, this "force-based" method was practically infeasible: context windows were tiny at first. Early LLMs supported only about 4,000 tokens — not even enough for a novel, let alone a corporate knowledge base. RAG had to be used. However, modern models have significantly larger context windows: some support over a million tokens.

For scale: one million tokens is roughly 700,000 words. You could fit the entire Lord of the Rings trilogy into such a prompt and still have room for The Hobbit.

This sharp leap in capability forces a difficult architectural question: if we can simply copy all documentation into the model's context window, do we still need the infrastructure overhead of embedding models and vector stores? Is RAG becoming an unnecessary layer of complexity?

If we assume that any necessary data truly fits in the context window, then the operating principle reduces to one word: simplicity. Three arguments in favor of direct context loading as the optimal solution.

Argument one: infrastructure simplification.

A production RAG system is serious engineering overhead. You need a chunking strategy: fixed size, sliding window, or recursive splitting? You will need an embedding model to encode data, a vector database to store it, and a reranker to sort results. Vectors must be continuously synchronized with source data. The result is many moving parts and points of failure.
Long Context offers what you might call a "stackless stack": remove the database, embeddings, and retrieval logic. The architecture reduces to a simple task: get the data and send it to the model.

Argument two: the retrieval lottery.

RAG introduces a critical point of failure — the retrieval stage itself. When a user asks a question, the system works with mathematical representations of data — vectors, sequences of numbers in an array — and tries to find the closest match. That is semantic search. But it is probabilistic by nature. For many reasons, retrieval may fail to find a relevant document. This phenomenon even has a name — "silent failure": the answer existed in the data, but the model never saw it because retrieval returned the wrong results. In the Long Context approach, there is no retrieval stage at all: the model sees everything.

Argument three: the "whole book" problem and the completeness bias.

RAG is fundamentally designed to retrieve what already exists: it searches for semantic correspondence between a query and a specific text fragment in the database. But what if the answer lies in what is not in the database?
Imagine you have a product requirements document and a separate release notes document. You ask: "Which security requirements were missed in the final release?"

With RAG, vector search will find fragments mentioning "security" and "requirements," retrieve excerpts from both documents — but it cannot retrieve the gap between them. RAG shows the model only isolated "snapshots," not giving it the holistic picture needed to identify omissions. To perform the comparison, the model truly needs both documents in full — exactly what Long Context provides by loading "the whole book": complete requirements and complete release notes into the context window.

Does this mean RAG is dead and vector databases belong in a 2024 technology museum? Not quite. Despite the advantages of Long Context in simplicity, RAG still has its place. Here are three more arguments in its favor.

Argument one: re-reading text.

Long Context creates serious computational inefficiency. Take, for example, a 500-page manual — roughly 250,000 tokens. If you place this document in the prompt on every user query, the model is forced to process it again each time.
RAG must also process the manual, but it pays that cost only once — at indexing time. Prompt caching can partially offset the cost for static data, but for dynamic streams where content changes frequently, you pay the full "price" on every request.

Argument two: the needle-in-a-haystack problem.

Intuitively, it seems that if data is in the context window, the model will necessarily use it. However, research shows the opposite. As context grows — say, to 500,000 tokens — the model's attention mechanism can "blur." If you ask a specific question about one paragraph buried somewhere in the middle of a 2,000-page document, the model will often either fail to find it or start "hallucinating," inventing details from surrounding text.
In RAG, we feed the model less "noise": by retrieving, say, only the top 5 most relevant fragments, we remove the haystack and leave only the needles. This forces the model to focus on signal, not background.

Argument three: infinite data volume.

A context window of millions of tokens sounds impressive, but at corporate data scale it is a drop in the ocean. A corporate data lake is typically measured in terabytes, or even petabytes. If you want to work with a truly unbounded dataset, you genuinely need a retrieval layer that filters information down to a volume that fits in the LLM's context window.

***

So where do we end up?

If your task works with a limited dataset and requires complex global analysis — for example, studying a specific legal contract or summarizing a book — I believe Long Context will be the optimal choice. It simplifies the technology stack and improves the quality of the model's reasoning.

But if you have to search across the infinite space of corporate knowledge, a vector database remains the only viable store for your data.