Per-annotator quality beats per-batch QA

The standard quality process for an annotation programme is to sample a batch, score the sample, and accept or reject the batch on the result. It is simple, it is auditable, and it is measuring the wrong unit.

What a batch score cannot tell you

A batch that scores ninety-two percent has told you that eight percent of it is wrong. It has not told you whether those errors came from one person who has misunderstood a guideline, from every person struggling with the same ambiguous category, or from a genuinely hard week of source material.

Those three causes need three different responses. The first needs a conversation and a re-review of that person’s prior work. The second needs the guideline rewritten, because the annotators are not the problem. The third needs nothing at all except a note that the batch was hard. A batch score cannot distinguish them, so in practice the response to a failed batch is to re-do the batch, which fixes those particular records and changes nothing about the next one.

Measuring the person instead

The alternative is to hold the score against the annotator rather than the delivery. Every reviewer sees a mix of work, blind, with a known proportion of items whose correct label is already established. Each annotator accumulates a rolling agreement rate, broken down by category.

What that structure gives you is the ability to distinguish:

• An individual who is drifting on one category — visible as one person’s rate falling while the cohort holds steady.
• A guideline that is genuinely ambiguous — visible as the whole cohort scoring low on the same category, which is a documentation defect and not a performance one.
• A source change — visible as every rate moving at once, on the date a new data source came online.
• A new starter still climbing — visible as a rate that is low but improving on the expected curve, which needs patience rather than intervention.

Only the first of those is an individual performance issue, and it is the one a batch-level process is least able to see.

Why this needs permanent people

A rolling per-person quality history is only worth building if the person is still there next quarter. On a task-queue model, where work is distributed to whoever is available, the unit being measured disappears between batches. You can score the output, but you cannot improve the source of it, so quality management collapses back into re-doing rejected batches forever.

This is the practical argument for permanent annotators, and it is an operational one rather than an ideological one. The compounding asset in a data programme is not the labelled records. It is a group of people who have internalised a domain’s edge cases, whose individual failure modes are known, and whose disagreements are informative because they are experienced. That asset takes months to build and does not survive being rebuilt every engagement.

Disagreement is a signal, not noise

The most useful output of per-person measurement is not the ranking. It is the set of items where experienced annotators disagree with each other.

Those items are the ones where the guideline is underdetermined, and they are almost exactly the items a model will get wrong later. Routing them to a domain reviewer, resolving them, and folding the resolution back into the guideline is the loop that improves a dataset rather than merely cleaning it. Batch sampling does not surface this set at all, because a disagreement between two competent people looks the same in a sample as a single person’s mistake.

What it costs

Per-annotator measurement needs a proportion of work to be double-labelled, which is a direct throughput cost, and it needs review capacity from people who actually know the domain. It is more expensive per record than sampling a batch.

It is cheaper per usable record, which is the figure that matters, because the errors it catches are the ones that would otherwise be discovered downstream — in an evaluation set, in a model that has learned a systematic mistake, or by a reviewer reading an output that a programme has already acted on.