Enterprise RAG: Why the Hard Part Is Not the Pattern, It's the Corpus

RAG, retrieval-augmented generation, is a simple pattern to state: before the model answers, the system retrieves the passages of your documents relevant to the question and hands them over as context. The model then answers based on your specific information — your contracts, your procedures, your history — not only on what it learned during training. It is the standard way to make an AI talk about what happens inside your company.

The pattern has been understood for a while and a demo takes days to stand up. That is why the interesting part for a company is not the pattern, but the corpus. At lab scale, RAG works with a hundred clean PDFs; at company scale it runs into per-document permissions, information that expires, sources of uneven quality, and answers you have to evaluate before you trust them. That is where projects stall, and that is where the attention belongs.

What RAG does, without the mystique

The mechanism has three beats. First, your documents get indexed: chopped into passages and stored so they can be searched by meaning, not just by exact word. Second, when a question arrives, the system retrieves the most relevant passages from that index. Third, those passages are handed to the model alongside the question, and the model writes the answer leaning on them.

The promise is twofold: the model answers with information it did not have — yours, current — and it can cite where it came from, which makes the answer verifiable. When it works, it replaces “check the manual” with a direct answer, source alongside.

The characteristic failure is also twofold, and worth naming: if the system retrieves the wrong passage, the model confidently writes an answer grounded in the document that did not apply. Retrieval is the silent half of the system. Almost nobody looks at it, and it is the half that decides whether RAG helps or lies with poise.

Why at company scale the problem is the corpus

A RAG demo and a RAG in production resemble each other the way a scale model resembles a building. The difference is not in the model or the pattern, but in the real state of your documents. Four fronts hold the work.

Per-document permissions. In a company, who can see what is not a detail, but an obligation. A RAG system that retrieves from the whole corpus without respecting the asker’s permissions can leak a payslip, a confidential contract, or a competitor’s price to someone who should not see it. Access control has to live inside retrieval, not as a cosmetic layer on top. This ties into AI agents and data privacy and it is not optional.

Freshness. A corpus that is not kept current ages and starts answering with the old version of the procedure, the repealed rate, the policy that changed. An obsolete document the system presents with the same confidence as a current one is worse than no answer, because it looks authoritative. Keeping the index up to date is continuous work, not an upfront chore.

Source quality. RAG retrieves what is there. If your documents contradict each other, mix drafts with final versions, or carry errors, the system retrieves them just as diligently. AI does not fix a messy archive, it only surfaces the mess faster. Much of the work before a useful RAG is, in truth, putting the corpus in order.

Answer evaluation. Without a way to measure whether answers are correct and well grounded, there is no serious way to say the system works. You need test cases with known answers, failure review, and quality thresholds. It is the same rigor I demand before giving any agent real work, and its absence is one of the causes I analyze in why enterprise AI pilots fail.

Front	The question you have to answer
Permissions	Does retrieval respect what each person is allowed to see?
Freshness	Does the index reflect the current version of each document?
Quality	Is the source in order, or does the system propagate the mess?
Evaluation	How do I know, with data, that the answers are correct?

Where the documents come from: RAG and the rest of the architecture

RAG solves one specific question: how the model answers based on your information. It does not solve how it reaches that information in the first place, which is a connection problem: the agent needs to get to your document manager, your ERP or your wiki, and that connection is better standardized than hand-wired. That is what the Model Context Protocol (MCP) is about, and it can be exactly the route by which the system reaches the corpus.

Nor is it the same as the agent’s memory. RAG retrieves from a stable body of documents. Memory and context in agents are about what the agent retains from a conversation or a case over time. They look alike because both “bring information to the model”, but one queries an archive and the other remembers its own. How these pieces fit into a complete system I develop in AI agent architecture.

When RAG is enough and when you need an agent

This is the distinction that saves a board the most, because it decides the scope of the project.

RAG, on its own, answers questions. It is the right tool when the work is to know: check a policy, find the applicable clause, summarize what your documents say about a matter. The person asks, the system answers with its source, and the person decides and acts.

An agent does more: beyond answering, it acts. When the work does not end in the answer but in an action — apply the policy to the case, generate the document, update the system, escalate the exception to a person — RAG is just a component inside something larger. The agent uses retrieval as one of its capabilities, but it also decides steps, touches systems, and leaves a trace of what it does. That distinction between answering and executing is what separates a smart search from an agent that works, and I develop it in the pillar on AI agents for business.

Put briefly: if your problem is that people cannot find what the company already knows, start with RAG done well. If your problem is that a whole process eats hours of repetitive work that still needs judgment, RAG is a piece, not the solution. Putting the corpus in order so retrieval is reliable is, in either case, part of treating the operating model as code: making explicit and usable what today lives scattered.

Frequently Asked Questions

What is RAG in a company?

It is a pattern that makes an AI model answer based on your own documents: before responding, the system retrieves the relevant passages of your information and hands them over as context. The answer then leans on your contracts, procedures or history, not only on what the model learned during training.

Why do RAG projects fail in production?

Rarely because of the model. They fail because of the corpus: retrieval that ignores each person’s permissions, obsolete documents presented as current, contradictory or messy sources, and no evaluation to measure whether answers are correct. The demo works with clean documents. The real operation lives on the corpus as it actually is.

RAG or fine-tuning the model on my data?

They are different things and get combined badly when confused. RAG brings fresh information at answer time and lets you cite the source, which is what you want when your data changes. Fine-tuning changes a model’s behavior, not its access to updated information. To answer about documents that change, RAG is almost always the starting point.

Does RAG replace an agent?

No: they solve different problems. RAG answers questions by leaning on your documents. An agent, beyond answering, executes a process: it decides steps, acts on your systems, and escalates what exceeds its mandate. When the work ends in an action rather than an answer, RAG is a component of the agent, not its replacement.