June 12, 2026 · 9 min read · llm · retrieval

Why your RAG answers confidently and wrongly

Nearly every failing RAG system I have reviewed was not broken in the model. The model did exactly what it does well: fluently retell what it was handed. The problem is that it was handed the wrong context — and nobody noticed, because the answer sounded convincing.

Where the answer breaks

Break the pipeline into four steps: chunking, retrieval, context assembly, generation. Log all four. Nine times out of ten the right passage never made the top 5 — and after that, no prompt is going to save you.

Retriever.cs

// log exactly what goes into the modelvar hits = await index.SearchAsync(query, top: 5);
logger.LogInformation("retrieved {Ids} score={Top:F3}",
    hits.Select(h => h.Id), hits.Max(h => h.Score));

if (hits.Max(h => h.Score) < 0.62)
    return Answer.Insufficient();  // better to say nothing

Refusing to answer is a feature. Users forgive “I don’t know.” They do not forgive a confident invention.

Retrieval beats prompting

Hybrid search (vectors plus BM25) almost always buys more than another paragraph of instructions. Then reranking. Only after that is it worth experimenting with wording.

Note

Chunk size follows the structure of the document, not advice from a blog post. For policy documents that usually means one section, not 512 tokens.

Caution

Never index documents with different access rights without a per-user filter. A leak through a model’s answer is still a leak.

Evals in CI

Fifty labeled questions, run on every pull request, recall@5 tracked separately from answer quality. That is enough to catch a regression before a customer reports it.

What to do on Monday

Turn on logging for what actually reaches the model. Collect twenty real user questions. Check how many of them retrieved the right passage at all. After that the fix is usually obvious.

Facing something similar?Get in touch