Compare
How to benchmark a document extraction API on your own documents
velrim-eval is a command-line tool, Apache-2.0, that scores a document extraction API against a labeled set of documents and turns the result into an exit code. It is the scorer behind Velrim's published comparison of six extraction setups (September 2026), and the comparison's raw outputs ship in the repository, so the same commands that grade a private corpus reproduce the published table. Five adapters: Velrim, OpenAI, Gemini, LlamaExtract and Mistral. There is no hosted service and no account. Every number it prints is the caller's result on the caller's documents.
Install
Node 20 or newer. npm install builds the CLI. On Windows, git config --global core.longpaths true before cloning, because some source documents have long file names.
Rescore a published result, no keys
Both commands run from the published raw outputs and make no network calls. The second prints the Mistral OCR 4 row of Velrim's published comparison of six extraction setups (September 2026): 40.3% [31.0, 52.5] on 96 absent fields, the same figure as the table on the write-up page.
Write the golden set
Three states per field. present carries the value that is on the page. null means the field is in the schema and the document does not contain it, so the right answer is null. missing means the label is unknown and the field is skipped. docClass is required, and schema is a JSON Schema file resolved relative to the golden file. An empty string counts as present, because the model did emit something and coercing it to null would hide a fabrication.
Run an adapter
- Keys come from the environment only:
VELRIM_API_KEY,OPENAI_API_KEY,GEMINI_API_KEY,LLAMA_CLOUD_API_KEY,MISTRAL_API_KEY. A missing key exits 2 before any network call. - Every live run prints a spend estimate and needs
--expected-spend-usd,--pricing-basis,--pricing-as-of, one frozen--cal-test-manifest <class>=<path>per document class, and--confirm-spend. Without the confirmation it makes zero calls. --repeat Nwrites one predictions file per repeat. Every row records whether the call completed, hit a transport failure or a contract failure. Failures stay in the file as empty predictions rather than disappearing.- Progress is checkpointed after every document-repeat, so an interrupted run resumes without paying again for finished units.
- Without
--live,runreads recorded responses and proves the pipeline is wired, not how a model performs.
Score it
score writes per-field precision, recall and F1 with a corpus micro-average, plus ECE, Brier and AUROC when the adapter returns a confidence per field. fabrication counts every absent field answered with any value as one fabrication, and answering null, leaving the key out, or returning an empty string or an "n/a" as declining. The judge is a comparison between the answer key and the output, no model and no confidence score involved. The full definition is on the metric page.
Gate the build
The build fails when corpus F1 drops below --min-f1, corpus ECE rises above --max-ece, or, with a baseline, either metric regresses by more than 0.005. The thresholds are the caller's.
Commands
| Command | What it does | Exit code |
|---|---|---|
run | Runs an adapter over a golden set and writes per-repeat predictions plus health and meta files | 0, non-zero on error |
score | Compares predictions with the golden set and writes scores.json, per document and corpus micro-average. No model calls | 0, non-zero on malformed input |
fabrication | Judges predictions for fabrication on absent fields and writes fabrication.json: the pooled rate with its interval, the strict and all-attempted rules, the answered-when-present rate, per-class rows. No model calls | 0, 2 on malformed input |
report | Renders per-field precision, recall and F1, the error curve and the risk-coverage curve, ECE, AUROC and Brier. --baseline adds a delta | 0, 2 on missing input |
ci | Gates scores.json against --min-f1 and --max-ece, with an optional --baseline no-regression check | 0 pass, 1 fail, 2 usage or IO |
calibrate | Fits a 1-D Platt curve on the (confidence, correct) points in a scores.json and writes the curve and a selective-prediction threshold | 0, 3 on absent or empty input |
The scoring math is the published @velrim/scoring package. Velrim's own error curves run the same code, and the CLI imports it by its published name, so there is no second copy to drift. The corpus, the frozen plan, the disclosures and the right-of-reply policy for the comparison are in the repository.