RAG that survives contact with production
Retrieval demos are easy. The gap between a convincing notebook and a system your support team trusts is mostly unglamorous engineering — here is where it usually goes wrong.
A retrieval demo takes an afternoon. A retrieval system your support team will actually lean on takes considerably longer, and almost none of the extra time goes into the model.
Here is the pattern we keep running into.
The demo lies about your documents
Demos are built on clean documents. Real corpora are exports: PDFs with two-column layouts, spreadsheets someone flattened into prose, a decade of wiki pages where half the content is stale and nothing says so.
Before touching an embedding model, spend a day reading a random sample of fifty documents from the actual corpus. You will find things no pipeline can fix on its own — duplicated policies with different effective dates, tables that lose meaning when linearised, sections that only make sense next to a diagram.
That day changes what you build. It usually adds a cleaning step, and it often shrinks the corpus dramatically.
Chunking beats model choice
Teams spend weeks comparing embedding models and minutes deciding how to split text. That is backwards. Swapping a good embedding model for another good one moves retrieval quality by a few points. Getting chunking wrong costs far more.
Two rules carry most of the benefit:
- Split on structure, not character count. A heading and its body belong together. A table belongs with its caption.
- Keep a parent reference. Retrieve the small chunk, feed the model the larger section it came from. Precision on the way in, context on the way out.
Measure the two halves separately
"The answer was wrong" is not a debuggable observation. There are two failure modes hiding inside it, and they need different fixes.
Build a small evaluation set — a hundred real questions with the document that should answer them — and track:
- Retrieval recall. Was the right chunk in the top k at all? If not, no prompt will save you.
- Answer faithfulness. Given the right chunk, did the model use it, and only it?
A hundred examples is enough to catch regressions. Teams skip this because it feels like overhead in week one; it is the only reason week twelve is calm.
Ship the refusal path first
The most damaging failure is not a wrong answer. It is a confident wrong answer that nobody catches because the system has never once said "I don't know."
Decide early what happens when retrieval comes back weak: a confidence threshold, a fallback to search results, a handoff to a person. Build that path before the happy path, and the first time the system is asked something outside its corpus it will do something sensible instead of improvising.
What this looks like in practice
On a recent engagement the first honest evaluation put retrieval recall at 61%. Nothing about the model changed. Restructuring chunks around document headings, de-duplicating superseded policies, and adding a parent-section lookup took it to 89% in under two weeks.
The model was never the problem. It rarely is.
Got a problem that looks like one of these?
Tell us what you are trying to build or automate. An engineer reads every enquiry personally.