Fabrication on absent fields: how the metric is defined and scored

A field is absent when the answer key says the document does not carry it. In the three-state golden format velrim-eval reads, a field is present with a value, null, or missing. The judge treats null and missing alike as absent. Present fields are scored by the accuracy metrics and never enter this one.

Round 1 has 2,102 labeled fields across 124 documents of four types. 142 of them were labeled absent by the original annotators, and every one of those is a missing cell.

One absent field answered with a substantive value is one fabrication. Declining to answer is not one. Five forms of declining are fixed in advance: an explicit null, an omitted key, an empty string, a whitespace-only string, and one of the tokens n/a, not present, not applicable, none and unknown, matched without regard to case or spacing.

One exclusion applies per field. A listed token that appears as a real value for that field anywhere in the answer key counts as substantive for that field. A literal none in a discount column is a value, not a refusal.

A strict variant treats every value other than null or an omitted key as a fabrication, so an empty string counts. velrim-eval computes both. The headline is the substantive rule, because it keeps the sentence "returned a value" true of every counted cell.

The judge is a function of the golden state and the output value, nothing else. It is a mechanical comparison, the same for every system. No model reads the output, no parsing or anchoring from any vendor takes part, and the confidence score is not consulted. Every count re-derives from the published predictions.jsonl files and the answer key, and the code is src/fabrication/judge.ts in the repository.

Each absent field on each document is one cell. When a system runs more than once over the same document, the cell's value is the mean over its repeats, so the headline stays a per-call rate. The rate is the mean over cells, pooled across document types. Round 1 ran every system three times.

The primary denominator holds every absent cell on a document whose call completed. A document that failed outright contributes nothing, so a system's rate is never lowered by its own outages.

The all-attempted rule is the companion. Every attempted document stays in, and a failed call scores as declining. Both rates print side by side with the completed/attempted count, so the interaction is shown instead of discovered. In round 1 the two Gemini setups completed 328 and 315 of 372 attempts, and their all-attempted rates sit below their primary rates for that reason.

A rate for a single document type prints only where at least 20 absent cells survive. Below that the row carries the count and points at the probe table.

The headline is pooled across document types rather than averaged per type, because the absent cells are spread unevenly. In round 1, after the label audit, receipts hold 63 of the 96 cells and one document type holds none.

Absent labels deserve less trust than present ones. A present label can be checked against the value on the page. An absent label claims a negative, and annotators skip fields more often than they invent them. Hand-checking every absent label against the page is part of the metric, not a preliminary step.

In round 1 all 142 absent labels were checked before any competitor ran. 40 were wrong: the value is printed in the document. 6 could not be settled: the region is illegible or redacted. All 46 were struck from the denominator for every system identically, and the golden files were left byte for byte unchanged. 96 labels remained. The per-label record is public, and a label shown wrong after publication is struck the same way, with the count disclosed.

Two document types in round 1 have almost no naturally absent fields. For those the metric has a second instrument: trap fields, called probes. A probe is a field taken from a sibling document type's schema that cannot occur in the target type. Three per type were drawn with a published seed and no discretion. Each probe is added to a copy of the schema, its answer is missing on every document, and the system runs once more over that copy.

A probe is struck when its value turns out to be visible on any page. Round 1 checked all 372 probe-document pairs visually before the plan was hashed and struck four probes. 198 probe cells survived per system.

Probes measure something narrower than natural absence. A natural absent field is plausible for the type and merely not filled in. A trap field is implausible for the type, and the system has the schema as its only reason to answer. The two numbers print in separate tables and are never pooled. Probes never touch the accuracy numbers, since accuracy is scored on the untouched schema.

Fields cluster within documents. A scan that defeats one field tends to defeat its neighbors, so the resampling unit is the document, never the field. Every bracket is a 95% interval from a bootstrap over documents, 10,000 resamples, bias-corrected and accelerated, with the seed published.

The intervals are wide because the cells are concentrated. 63 of the 96 natural absent cells in round 1 sit in 15 receipts. A field-level bootstrap would understate every interval.

The reading rule is fixed in advance. Where two systems' ranges overlap, this test cannot tell them apart, and no winner is named. Round 1 pooled 17% across the six systems on natural absence. Five of the six overlap one another, and only the Mistral OCR 4 range sits clear of the rest.

The answered-when-present column sits in the same table on purpose. A system that declines more fabricates less by construction, so the cost of caution prints beside the rate it buys. In round 1 Velrim leaves about 12% of present fields blank against 3% to 4% for the others.

velrim-eval computes the metric from a predictions file and a golden file. The golden format has three states per field:

missing and null are the absent states. A present field carries a value. An empty string in a prediction is scored as present by the accuracy path and as declining by the headline fabrication rule, because it is not a substantive value.

Run any adapter over the golden set, then judge the predictions:

fabrication.json holds the pooled rate with its interval, the strict rule, the all-attempted rule, the answered-when-present rate, the completed/attempted count, the per-type rows, and the mean confidence on the system's own fabrications where the output carries one. To reproduce a round-1 row, point the same command at the published outputs and the strike record:

Neither command needs an API key or spends money. Only run --live does, and it prints the expected cost first.

The metric and its round-1 numbers are archived under DOI 10.5281/zenodo.22233430. The repository is github.com/velrimhq/velrim-eval, the plan frozen before the first paid call is ANALYSIS-PLAN.md there, and the round-1 write-up is on this site.

The per-field outputs, the probe set and the label audit are also published as a Hugging Face dataset, velrim/fabrication-on-absent-fields.

Velrim (2026). Fabrication on absent fields: a pre-registered six-arm document-extraction bake-off (round 1). Zenodo. https://doi.org/10.5281/zenodo.22233430