What Is RAG, in Plain English — and When Does a Business Actually Need One?

Picture a new hire who has read most of the internet and none of your company. Genuinely brilliant, remembers a staggering amount, and has never seen a single document you own.

Ask them a general question and they are excellent. Ask them what your refund policy says and they will produce something plausible, confident, and invented.

RAG — Retrieval-Augmented Generation — is what happens when you give that person access to the filing cabinet before they answer. They look up the relevant pages first, then answer using what they found. That is the whole idea. The name is much bigger than the concept.

The two halves, and why they are separate

A RAG system does two things in order.

Retrieval. Given a question, find the passage in your documents that answers it. This is a search problem wearing a new hat, and the quality of the search determines almost everything downstream.

Generation. Hand that passage to the model and let it write the answer, grounded in what was actually retrieved rather than in what it remembers from training.

Keeping those halves separate is not pedantry, it is the entire practice. When a RAG system gives a wrong answer, the first question is always which half failed. Did the wrong passage come back, or did the right passage come back and the model ignored it? Those two failures have completely different fixes, and a team that does not measure them separately will spend months tuning the wrong one.

Why not just put everything in the prompt

Because you cannot. A company knowledge base is millions of words; a prompt is thousands. Even as context windows grow, dumping everything is slow, expensive, and measurably worse — models attend less carefully to the middle of a very long context than to its edges, so more documents can mean worse answers.

Retrieval is how you send the model the two pages that matter instead of the two thousand that do not.

The part everyone gets wrong

The most important behaviour in a RAG system is not how good it is when it knows the answer. It is what it does when it does not.

The correct behaviour is "I could not find that in your documents." The wrong behaviour is the model filling the gap from general knowledge and sounding exactly as confident as when it was right. A system that is correct ninety-nine times out of a hundred and indistinguishable on the hundredth is worse than a system that admits the hundredth, because only one of them lets you build trust.

So when do you actually need one?

You need RAG when all three of these are true:

The answers live in documents you own. Policies, procedures, contracts, manuals, tickets, research. If the knowledge is not written down anywhere, no amount of retrieval will find it, and the real project is a documentation project.

The knowledge changes. Static knowledge can be fine-tuned in or written into a prompt. Knowledge that changes weekly cannot, because you would be retraining constantly. Retrieval picks up a new document immediately.

Being wrong has a cost. If a plausible-sounding wrong answer is merely annoying, you may not need the machinery. If it is a compliance problem, a refund, or a lost customer, you need answers that can be traced to a source.

If one of those is false, there is usually something simpler. A well-organised wiki with good search beats a badly-built RAG system every single time, and it costs a fraction of the money.

The honest failure mode

Most RAG projects that disappoint are not retrieval failures. They are corpus problems. Two versions of the same policy both indexed. A procedure superseded three years ago still sitting in the folder. A scanned PDF whose text layer is gibberish.

Retrieval will find those confidently and present them to the model as authority, which is worse than finding nothing. The single highest-return activity in a RAG build is usually deleting documents, not tuning embeddings.

We go deeper on the mechanics in Understanding RAG — this one is the version to send to a colleague who asked what the acronym means.

If all three tests above came back true, this is RAG systems work — and the first week of one is usually spent deleting documents rather than tuning embeddings.