Retrieval is a data problem before it is a model problem

A retrieval system that answers well in a demonstration and badly in production has usually not failed at the model. It has failed at the corpus, and the failure was present before anything was embedded.

This matters because the two failures look identical from the outside — a wrong answer, confidently phrased — and they have completely different fixes. Swapping the embedding model is the cheap thing to try, so it is what gets tried, and it rarely moves the number.

Four failures that live in the corpus

The document is in the index but was never readable

Scanned pages, tables that lost their structure in extraction, forms whose meaning is carried by layout rather than by text. These become chunks of plausible-looking prose with the relationships stripped out. Retrieval finds them, the model reads them, and the answer is wrong in a way that is very hard to trace, because the passage genuinely does appear to support it.

The corpus contains several answers and no way to choose

Regulated corpora accumulate superseded policy, draft versions, and guidance that applied to a programme that ended. Without effective-date and supersession metadata, retrieval treats a rescinded memo and the memo that replaced it as equally relevant, and relevance ranking has no basis on which to prefer the current one.

The chunk boundary destroyed the meaning

Fixed-size chunking splits a condition from its exception and a requirement from its scope. A retrieved chunk that says a thing is required, separated from the sentence three hundred tokens later saying when it is not, produces an answer that is confidently and precisely wrong.

The question and the corpus use different words

People ask in the vocabulary of their work. The corpus is written in the vocabulary of policy. Embeddings absorb some of that gap and not all of it, and the residue is concentrated in exactly the domain-specific terms that matter most.

Evaluate retrieval on its own

The single most useful change to a struggling retrieval system is to stop scoring it end to end. Build a set of questions with the passages that should be retrieved for each one marked by someone who knows the domain, and score the retrieval step alone.

This is ordinary ground-truth work, it is the part teams skip, and it separates two questions that an end-to-end score fuses together: did we put the right passage in front of the model, and did the model use it correctly. Until those are separated, every improvement is a guess. Once they are, the work is usually obvious — and it is usually extraction, metadata and chunking rather than anything to do with the model.

What good looks like before the model is involved

• Extraction is verified against the original document, including tables and scanned pages, rather than assumed to have worked.
• Every document carries effective date, supersession status and provenance as structured metadata, and retrieval filters on them.
• Chunking follows the document’s own structure — sections, clauses, list items — so a boundary never separates a rule from its scope.
• A retrieval evaluation set exists, is drawn from real questions, and is scored separately from generation.
• Access controls are applied at retrieval time, so a passage the asker may not read is never a candidate in the first place.

The last one is not an optimisation. In an environment with compartmented data, a retrieval layer that ranks across everything and filters afterwards has already leaked — the ranking itself is an observable signal about content the asker is not cleared to see.

The reframe

Retrieval work presents as a modelling problem because that is the part with interesting choices in it. In regulated environments it is mostly document engineering: getting text out of documents faithfully, attaching the metadata that says which version governs, cutting on boundaries the document itself defines, and building the evaluation set that tells you whether any of it worked. None of that is glamorous, and all of it is the difference between a system that demonstrates well and one that can be accredited.