Guide

Choose confidence thresholds for document extraction review

All guides

A confidence cutoff is a policy choice with a measurable tradeoff: which fields enter review, and how many errors remain among those that do not. Start by using scores to prioritize review. Automatic acceptance requires evidence for the exact document population, fields and decision rules you plan to deploy.

Velrim publishes calibrated confidence measurements, including reliability diagrams and risk–coverage curves by document class. Those artifacts describe their evaluated populations and calibrator versions. They do not establish an automatic-acceptance threshold for arbitrary customer documents. The absent-field study examines another failure mode you should include in your own evaluation.

Define the decision unit and non-score rules

Decide whether you are accepting a field, a whole invoice or a downstream action. A low error rate per field is not a document-level guarantee, and several correct values do not make a wrong bank account safe. Evaluate the same unit your production workflow acts on.

Before a score cutoff, apply required-field, conflict, source-evidence and business-rule checks. Missing values and absent scores do not pass by default. A verified grounding entry means the value was located in the cited region; it can still be the wrong value for the field. Write your policy down so the evaluation and production branch match.

Build a labeled set that includes failures

Sample the document sources, layouts and field roles you actually receive. Label answers against the original documents, including fields with no answer. Retain failed requests and missing outputs in the workflow-level analysis. If you only score successful present fields, you hide part of the review workload.

Use one set to choose a candidate threshold and a separate held-out set to estimate its behavior. Split by document, and where appropriate by template or customer, so nearly identical pages do not appear on both sides. Keep the held-out labels out of prompt and policy tuning. Report results per important field and document group as well as pooled totals.

Calculate empirical risk and coverage

For a field-level experiment, coverage is accepted fields divided by all evaluated fields. Empirical risk is incorrect accepted fields divided by accepted fields. If nothing is accepted, risk is undefined; it is not zero. These are observed proportions, not a certification or a bound on future error.

The following download evaluates an already-labeled policy input. eligible means all non-score checks passed; correct comes from ground truth; confidence is a number or null. Include ineligible rows in the denominator. Keep document and field IDs in your own dataset for grouping, even though this small calculator only reads these three columns. Download thresholds.ts.

thresholds.ts · empirical field-level summary
import { readFile } from 'node:fs/promises';
import { pathToFileURL } from 'node:url';
// One labeled field per row, including omissions and rule failures.
// eligible means your non-score checks passed, not that the value is correct.
type Row = { confidence: number | null; eligible: boolean; correct: boolean };
export function summarize(input: unknown, threshold: number) {
if (!Number.isFinite(threshold) || threshold < 0 || threshold > 1) {
throw new Error('Threshold must be in [0, 1]');
}
if (!Array.isArray(input) || input.length === 0) throw new Error('Expected labeled rows');
const rows: Row[] = input.map((row: unknown) => {
if (typeof row !== 'object' || row === null) throw new Error('Invalid row');
const r = row as Record<string, unknown>;
if (
typeof r.correct !== 'boolean' ||
typeof r.eligible !== 'boolean' ||
(r.confidence !== null &&
(typeof r.confidence !== 'number' ||
!Number.isFinite(r.confidence) ||
r.confidence < 0 ||
r.confidence > 1))
) {
throw new Error('Each row needs correct, eligible and confidence (number or null)');
}
return { correct: r.correct, eligible: r.eligible, confidence: r.confidence };
});
const accepted = rows.filter(
(r) => r.eligible && r.confidence !== null && r.confidence >= threshold,
);
const errors = accepted.filter((r) => !r.correct).length;
return {
threshold,
total: rows.length,
accepted: accepted.length,
errors,
coverage: accepted.length / rows.length,
empiricalRisk: accepted.length === 0 ? null : errors / accepted.length,
};
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
const path = process.argv[2];
if (!path) throw new Error('Usage: npx tsx thresholds.ts labeled-fields.json');
const rows: unknown = JSON.parse(await readFile(path, 'utf8'));
// Candidate thresholds for exploration only; none is a recommended policy.
console.table([0.5, 0.7, 0.9, 0.95].map((t) => summarize(rows, t)));
}

Run the synthetic example locally

Install tsx as a development dependency in a Node 22+ project. Save both downloads in one directory and run npx tsx thresholds.ts labeled-fields.json. No API key or provider call is needed. Download labeled-fields.json contains six invented rows solely to demonstrate the arithmetic; they are not Velrim measurements.

Run the synthetic example locally
Candidate cutoffAccepted / totalErrors / acceptedInterpretation of this synthetic set
0.73 / 61 / 350% coverage; about 33.3% observed risk.
0.92 / 61 / 233.3% coverage; 50% observed risk.
0.951 / 60 / 1One correct field is too little evidence to justify a policy.
labeled-fields.json · synthetic teaching data
[
{ "confidence": 0.96, "eligible": true, "correct": true },
{ "confidence": 0.93, "eligible": true, "correct": false },
{ "confidence": 0.85, "eligible": true, "correct": true },
{ "confidence": 0.65, "eligible": true, "correct": false },
{ "confidence": null, "eligible": false, "correct": false },
{ "confidence": 0.99, "eligible": false, "correct": false }
]

Do not choose a policy from a flattering point estimate

The synthetic example intentionally gets worse between 0.7 and 0.9. A higher cutoff does not guarantee lower observed error in a finite sample. Inspect counts and mistakes, not just the plotted line. If no candidate meets your criteria, keep those fields in review.

For an automatic-acceptance decision, report uncertainty as well as observed risk. Fields within a document can share errors, so treating every field as an independent trial can overstate precision. Use an analysis that respects document grouping and your decision unit. This small calculator does not compute uncertainty intervals or select a production threshold.

Confirm the chosen policy once on held-out documents before rollout. Preserve the schema, policy, model and meta.calibrator_version with the evaluation. Recheck when those change, when document sources shift or when review reveals a new failure mode. Sample accepted outputs too: observing only reviewed fields cannot measure the errors escaping review.

Put the result into a review workflow

The TypeScript SDK’s result.belowThreshold(t) returns pointers below the chosen score plus missing fields. It is a convenience filter, not the whole policy: handle absent metadata, absent scores, nulls, conflicts and required fields explicitly. The n8n example starts with those non-score checks and keeps the source references for reviewers.

If you are starting from a plain extracted object, first retain field metadata in your integration. If you are changing providers, compare review behavior before cutover. A threshold becomes useful when it is attached to a tested workflow and a measured workload.

Written by Velrim.