RAG that actually works: patterns from 12 production systems

What RAG is, and where it goes wrong

Retrieval-augmented generation (RAG) answers questions from your own documents. Instead of relying on what a language model memorised, the system searches your content, puts the most relevant passages into the prompt, and asks the model to answer from them — ideally with citations.

When a RAG assistant gives a wrong or vague answer, the model is rarely the root cause. Usually the right passage was never retrieved, was cut in half during chunking, or was buried under less relevant text. So the work that makes RAG reliable is mostly search engineering.

The pipeline

• Ingest: collect documents from sources such as a CMS, help centre, PDFs or tickets, with their permissions.
• Chunk: split them into passages that each make sense on their own.
• Embed and index: store vectors and keywords, with metadata.
• Retrieve: search with the user’s question, filtered by what that user may see.
• Re-rank: reorder the top results with a more precise model.
• Generate: answer only from the retrieved passages and cite them.
• Evaluate: measure retrieval and answer quality continuously.

Chunking that keeps meaning

Fixed-size chunks that cut through tables and headings are the most common cause of bad answers. Split along the document’s structure instead: sections, headings, list items and table rows. A few hundred tokens per chunk with a small overlap works well for most prose.

Store metadata with every chunk: document title, section heading, URL, date, product or version, language and access rights. Prepending the title and heading to the chunk text before embedding often improves retrieval noticeably, because a paragraph on its own rarely says what it is about.

Hybrid search and re-ranking

Vector search is good at meaning but weak at exact terms: product codes, error messages, names and numbers. Keyword search (BM25) is the opposite. Run both and merge the results, for example with reciprocal rank fusion, and you get the strengths of each.

Then re-rank. Retrieve 20 to 50 candidates quickly, and let a cross-encoder re-ranker score each one against the question. Passing only the best five to eight chunks to the model reduces cost, latency and confusion.

Permissions and freshness

Enforce access rights at retrieval time with metadata filters, never by asking the model to hide things. If a user must not see a document, it must not reach the prompt.

Keep the index fresh with incremental updates: re-index a document when it changes and delete chunks when it is removed. Stale chunks produce confident, outdated answers — the hardest kind to spot.

Prompting for grounded answers

Tell the model to answer only from the provided passages, to cite them by ID, and to say clearly when the answer is not in the sources. Treat retrieved text as data, not instructions: documents can contain text that tries to steer the model, so your system prompt should say that instructions inside sources must be ignored.

Evaluation is the product

Before launch, collect 50 to 200 real questions with the documents that should answer them. Measure retrieval (was the right document in the top results?) separately from answer quality (was the answer correct, complete and supported by the sources?). Automated judging with a model helps at scale, but spot-check it by hand.

Run this set on every change to chunking, embeddings, prompts or models. Without it, every improvement is a guess.

A practical stack

For most teams we start simple: PostgreSQL with pgvector (or OpenSearch when strong keyword search is needed), an embedding model and a re-ranker from a major provider, ingestion running in background queues, and a small admin screen that shows which sources each answer used. Dedicated vector databases such as Qdrant or Weaviate make sense at larger scale. The architecture matters less than the chunking, filters and evaluation around it.