RAG Is Not a Feature

Teams ask for RAG like they are ordering a component. Add retrieval, ship the feature, move on.

It is not a component. It is a contract: given a question, the system promises to hand the model the passage that answers it. Everything difficult lives in whether that promise holds, and almost none of it is the embedding model.

The retrieval contract

Write it down explicitly. For every question the system must answer, there is a document and a passage that should come back. If you cannot write that down, you do not have a retrieval problem yet, you have a specification problem, and no amount of tuning will fix it.

This is why we build the question set before we build the pipeline. Twenty or fifty real questions, each with the passage that should be retrieved. Now you have something you can actually measure, and measurement is the part everyone skips.

Chunking is the part people obsess over

Chunk size, overlap, semantic splitting, late chunking. It matters, and it is maybe a tenth of the outcome. The debate is popular because it is easy to have without data.

Here is what beats any chunking strategy: cleaning the corpus first. Duplicate policies. A procedure superseded two years ago that is still indexed. Scanned PDFs where the text layer is garbage. A folder labelled "archive" that nobody excluded. Retrieval will find these confidently and hand them to the model as authority, which is worse than finding nothing.

Evaluation is the other nine tenths

The questions are answered, but did the right passage come back? Most teams never check. They check whether the final answer sounds good, which is a different question and a much weaker signal, because a fluent model can produce a plausible answer from the wrong passage and you will never know.

Track the retrieval separately from the generation. When the answer is wrong, you need to know which half failed. If retrieval returned the right passage and the model still got it wrong, that is a generation problem with a completely different fix.

Refusal is the default, not the fallback

The most important behaviour in a retrieval system is what it does when retrieval comes back weak.

The correct answer is "I could not find that in your documents." The wrong answer is the model filling the gap from its training data and sounding exactly as confident as when it was right. A system that is right ninety times out of a hundred and indistinguishable on the tenth is worse than one that says "I don't know" on the tenth, because only one of them lets you build trust.

That is the contract. Not "we added retrieval," but "we know which questions this fails on, and it tells you when it does."

It is also the first thing we ask for on a RAG build, because a pipeline tuned against no question set is an opinion with a latency number attached.