The Prototype Lied to You

A RAG prototype is a weekend project. You embed some documents, drop them in a vector store, retrieve the top matches, and stuff them into the prompt. It works, and it feels like magic.

Then you point it at ten thousand real documents and the magic gets patchy. Here is what actually matters once you are past the demo.

Retrieval Is the Whole Game

If the right chunk is not retrieved, no model can save the answer. Most bad RAG answers are retrieval failures wearing a generation costume.

What moves the needle:

• Chunking that respects the structure of the document, not fixed character counts
• Storing useful metadata so you can filter before you rank
• Hybrid search, mixing keyword and semantic, instead of trusting embeddings alone

Garbage In, Confident Garbage Out

The model will happily answer from whatever you give it, including stale or contradictory sources. Cleaning and deduplicating your source data does more for quality than swapping models.

You Cannot Improve What You Cannot Measure

Build an eval set of real questions with known good answers before you tune anything. Without it, every change is a vibe. With it, you can actually tell whether a new chunking strategy helped or hurt.

Cite Everything

Make the model show which source each claim came from. It keeps the model honest, it makes hallucinations obvious, and it lets users trust the answer because they can check it.

The Honest Summary

RAG is not hard because of the model. It is hard because it is a search problem, a data-quality problem, and an evaluation problem stacked together. Treat it that way and it becomes reliable. Treat it as "embed and pray" and it stays flaky.