Most RAG failures are retrieval failures, not generation failures. Getting the right passages into the context depends on chunking that respects document structure, hybrid keyword and vector search, a reranking step, and evaluation of retrieval separately from answer quality.
Retrieval is the hard part
A generation model can only work with what it is given. When an answer is wrong or vague, the cause is usually that the relevant passage was never retrieved, or was retrieved in a form that lost its context.
Separate the pipeline into retrieval and generation so each can be measured. Retrieval quality is measured by whether the correct passages appear in the top results, which is a testable property independent of the model.
This separation also lets you improve the system without touching the model, which is usually where the gains are. Reranking and better chunking typically outperform swapping to a larger model.
- Measure retrieval recall separately from answer quality.
- Instrument which chunks were used for each answer.
- Improve chunking and ranking before changing models.
- Build a test set of real questions with known source passages.
Chunking that preserves meaning
Fixed-size chunking splits sentences and tables arbitrarily, producing fragments that lose their referent. A paragraph that says it must be approved by the regional manager is useless without the heading that identifies what it refers to.
Chunk on document structure: headings, sections, list boundaries and table rows. Attach the heading path to each chunk so an isolated fragment still carries its context, and keep tables as structured units rather than flattened text.
Overlap adjacent chunks slightly to avoid losing information at boundaries, and store metadata such as document, section, version and effective date. Version and date matter in enterprise content, where the correct answer is often the most recent applicable policy.
| Approach | Strength | Weakness |
|---|---|---|
| Fixed token window | Simple, uniform | Splits meaning, loses structure |
| Recursive by separator | Respects paragraphs | Still ignores headings and tables |
| Structure aware | Preserves headings and lists | Requires parsing per format |
| Semantic clustering | Coherent topics | Variable size, more compute |
| Parent-child | Small chunks retrieved, larger context returned | Extra storage and mapping |
| Table aware | Preserves tabular relationships | Format-specific handling |
Hybrid search and reranking
Vector search matches meaning and copes with paraphrase, but it is weak on exact tokens. An engineer searching for error code E1042 or part number AX-3312 needs lexical matching, because embeddings treat those strings as near-arbitrary.
Hybrid search combines both and merges results, typically with reciprocal rank fusion. Adding metadata filters for date, department, document type and permission scope narrows the candidate set before ranking, which improves both speed and precision.
Reranking with a cross-encoder over the top candidates is the highest-value single improvement in most pipelines. Retrieval casts a wide net cheaply; the reranker then scores candidates against the actual query with far more precision.
- Combine lexical and vector retrieval and merge with rank fusion.
- Apply metadata filters before ranking where possible.
- Rerank the top candidates with a cross-encoder.
- Tune candidate count for the latency budget you can afford.
Permissions, freshness and operations
Retrieval must respect the requesting user's access. Filtering after generation is both a security hole and poor practice, because the model may have already incorporated content the user cannot see. Apply permission filters in the retrieval query itself.
Content freshness requires an ingestion pipeline that updates the index when source documents change and removes deleted documents promptly. Stale policy content produces confidently wrong answers, which is worse than no answer in a regulated context.
Finally, keep the answer grounded and attributable. Require citations to the retrieved passages, and provide the source so a user can verify. Systems that answer without attribution are trusted less and used less, regardless of accuracy.
- Apply permission filters inside the retrieval query.
- Update and delete index entries as sources change.
- Require citations to retrieved passages in every answer.
- Log retrieval results and answers for audit and evaluation.
Frequently asked questions
Why does my RAG system give wrong answers?
Usually because retrieval failed. If the correct passage is not in the retrieved context, the model cannot answer correctly. Evaluate retrieval recall separately from answer quality before tuning prompts or changing models.
Do I need a vector database for RAG?
Not necessarily. Many databases now support vector indexes alongside relational and full-text search, which simplifies operations and makes hybrid search easier to implement in a single system.
How do you stop RAG leaking data between users?
Apply authorisation filters inside the retrieval query so the search itself can only return documents the requesting user may access. Filtering after generation is insecure because the model may already have used the content.
References
Engineer your next strategic platform
Estimate your project investment and delivery timeline instantly, or schedule an architectural discovery call with our engineering leadership.