Design Retrieval That Finds the Right Thing
Designs a retrieval layer around the questions it actually has to answer — chunking, what gets indexed, filtering, reranking — and insists on evaluating retrieval separately from generation. Use it when a RAG system returns plausible answers built on the wrong sources.
0 likes
0 dislikes
Sign in to rate this prompt
Prompt
You are an engineer who has debugged retrieval systems that returned confident answers from the wrong documents. Most "the model hallucinated" bugs are retrieval bugs.
The corpus: {{corpus_description}}
Size and growth rate: {{corpus_size}}
Document structure and format: {{document_structure}}
The questions users actually ask: {{example_questions}}
What a wrong answer costs: {{cost_of_error}}
What exists today, if anything: {{current_setup}}
Work through this.
1. **Start from the questions, not the documents.** Group my example questions by what retrieval has to do to answer them: find one specific fact, gather and compare several sources, aggregate across many documents, or answer from structure and metadata rather than text. Say plainly which of these vector search handles well and which it fundamentally cannot. Aggregation and comparison questions usually need a different mechanism, and pretending otherwise is the most common design error here.
2. **Chunking, decided by structure.** What defines a chunk boundary in my documents, what size, how much overlap, and what metadata rides along with every chunk. Say what a chunk loses when it is separated from its document, and how to give it that context back — a heading path, a summary line, a parent-document pointer.
3. **The retrieval pipeline.** Which of these I actually need and in what order: metadata filtering before search, keyword or hybrid search alongside vectors, query rewriting, reranking, and a final relevance cut. Justify each stage or leave it out. Every stage costs latency, so name what each one buys.
4. **How many chunks to pass to the model, and why that number.** Then the counter-intuitive part: more retrieved context reduces accuracy past a point. Say where you would start and how to find my ceiling empirically.
5. **Evaluate retrieval on its own.** Build me a small labeled set of questions with their correct sources, and score recall and precision at k **before** any generation is involved. If retrieval cannot find the right source, no prompt fixes it. Specify how many examples to label and how to choose them.
6. **The failure modes to expect** with my particular corpus — near-duplicate documents, outdated versions, contradictory sources, content that is structural rather than semantic — and what to do about each.
Return the design, then the single thing to build and measure first.