Your accuracy number is not an evaluation

Most systems arrive with one number attached. Ninety-four percent. It is the number in the pilot deck, the number in the status report, and the number a programme office is asked to accept as evidence that something is ready to deploy. It is almost never evidence of anything useful.

The problem is not that the number is wrong. It is usually computed correctly. The problem is that it is an average over a test set, and an average is the one summary guaranteed to hide the thing you need to see.

Averages hide the cases that decide deployment

Take a document classification system running at ninety-four percent. That figure is compatible with all of the following, and it does not distinguish between them:

• The six percent of errors are spread evenly and randomly across every document type.
• The six percent are concentrated entirely in one category that happens to be the highest-consequence one.
• The system is near-perfect on the eighty percent of traffic that is routine, and close to a coin flip on the long tail where a human would have escalated.
• The system is right for the wrong reason on a class of inputs that will change format next quarter.

Only the first is acceptable, and it is the least likely. Error concentrates. It concentrates in rare document types, in the languages with the least training data, in scanned pages rather than native text, and in exactly the ambiguous cases a person would have flagged. Those are also the cases an accreditation review will ask about, because they are the ones where an error has a consequence.

What to measure instead

The replacement is not a bigger test set. It is a set of slices chosen because a decision depends on each one, scored separately, and reported separately. In practice that means answering four questions the aggregate cannot.

Where does it fail, not how often

Break performance down by the dimensions your programme actually distinguishes: document type, source system, language, input quality, and the categories that carry different consequences. A slice with fifty examples and sixty percent accuracy tells you more than a slice with fifty thousand at ninety-four.

What does it do when it should decline

A system that answers everything is not more useful than one that declines the cases it cannot handle — it is more dangerous, because the failure is silent. Refusal behaviour deserves its own evaluation set: inputs that are out of scope, inputs that are adversarial, and inputs where the correct answer is that the record is insufficient. Measure whether the system declines them, and measure how often it declines things it should have answered.

Is it right for the right reason

On a retrieval-backed system, a correct answer drawn from an irrelevant passage is a failure that has not happened yet. Score the retrieval step separately from the generation step. A pipeline that scores well end to end while retrieving badly will fail the moment the corpus changes.

What happens at the edges of the distribution

Evaluation sets drawn from curated examples measure the middle of the distribution. Evaluation sets drawn from real traffic measure what the system will meet. They are not the same set, and the difference is where deployment risk lives.

The reporting change matters as much as the measurement

A breakdown that gets averaged back into one figure before it reaches the people making the deployment decision has not helped anyone. The output of an evaluation should be a table a reviewer can read down: slice, size, score, and what a failure in that slice would cost. That format survives contact with a review board, because it answers the question a board actually asks, which is never "how accurate is it" but "what happens when it is wrong, and how often is it wrong in that particular way".

It also changes what the engineering team does next. One number tells you to improve the model. A breakdown tells you which slice to build ground truth for, and that is usually a data task rather than a modelling one.

The uncomfortable part

Slicing an evaluation almost always makes the headline figure worse, because it stops averaging strong performance on easy traffic against weak performance on hard traffic. That is the point. A system whose weaknesses are known and bounded can be deployed with controls around it. A system with one impressive number and no breakdown cannot be deployed safely, whatever the number says — it can only be deployed hopefully.