Compare

OpenAI structured outputs from PDFs: the request, the price per page, and the measured fabrication rate

gpt-5.4-mini takes a PDF as a file part on Chat Completions and returns JSON. This page has the exact request Velrim's published comparison of six extraction setups (September 2026) sent it, free-decode and with Structured Outputs, OpenAI's docs on file inputs, schema rules and prices as quoted on the day, and the model's numbers on 124 public documents: accuracy, how often it invented a value for a field the document did not contain, how often it answered a planted field that could not exist, and the difference constrained decoding made. This setup beat Velrim on accuracy in that comparison, and the table says so.

The request

One Chat Completions call, the PDF as a base64 file part, the schema in the prompt text, temperature 0, model pinned to the dated snapshot. This is the body Velrim's published comparison of six extraction setups (September 2026) sent for every document, with a two-field schema in place of the corpus schemas.

terminal
curl -s "https://api.openai.com/v1/chat/completions" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d @request.json
request.json
{
"model": "gpt-5.4-mini-2026-03-17",
"temperature": 0,
"messages": [{
"role": "user",
"content": [
{ "type": "file", "file": { "filename": "document.pdf", "file_data": "data:application/pdf;base64,<base64 of the PDF>" } },
{ "type": "text", "text": "Extract the following fields from the document. Return JSON matching this schema. Use null for fields not present in the document.\n\nJSON Schema:\n{\"type\":\"object\",\"properties\":{\"invoice_number\":{\"type\":\"string\"},\"total\":{\"type\":\"number\"}},\"required\":[\"invoice_number\",\"total\"]}" }
]
}]
}

The prompt text is the sentence every model in the comparison got, "Extract the following fields from the document. Return JSON matching this schema. Use null for fields not present in the document.", followed by the JSON Schema. The frozen body also carried logprobs: true until the smoke run, when the API declined it with a 403 on a PDF file input. The parameter was removed and the trim recorded in the run manifest, which is why the confidence column for this row reads "none surfaced". The record is in the analysis plan, §12.

Structured Outputs: one more field

The structured variant sends the identical prompt and file and adds a response_format block with the same schema. The model then cannot emit a token that breaks the schema.

request.json, response_format
"response_format": {
"type": "json_schema",
"json_schema": { "name": "extraction", "strict": true, "schema": {"type":"object","properties":{"invoice_number":{"type":"string"},"total":{"type":"number"}},"required":["invoice_number","total"]} }
}
Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value.
OpenAI docs, Structured Outputs, read
Structured Outputs is the evolution of JSON mode. While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence. Both Structured Outputs and JSON mode are supported in the Responses API, Chat Completions API, Assistants API, Fine-tuning API and Batch API.
OpenAI docs, Structured Outputs, read
To use Structured Outputs, all fields or function parameters must be specified as required. Although all fields must be required (and the model will return a value for each parameter), it is possible to emulate an optional parameter by using a union type with null.
OpenAI docs, Structured Outputs, read
Structured Outputs only supports generating specified keys / values, so we require developers to set additionalProperties: false to opt into Structured Outputs.
OpenAI docs, Structured Outputs, read
When using Structured Outputs with user-generated input, OpenAI models may occasionally refuse to fulfill the request for safety reasons. Since a refusal does not necessarily follow the schema you have supplied in response_format, the API response will include a new field called refusal to indicate that the model refused to fulfill the request.
OpenAI docs, Structured Outputs, read

The comparison's frozen schemas list every field as required, set additionalProperties to false and keep every leaf nullable, which is how the docs say to express a field that may be absent under strict: true. The same schema bytes went to both rows, so the only difference between them is the response_format block. One corpus schema, cut to three fields:

cord-v2.schema.json, three of nine fields
{
"type": "object",
"properties": {
"subtotal": { "type": ["string", "null"] },
"tax": { "type": ["string", "null"] },
"total": { "type": ["string", "null"] }
},
"required": ["subtotal", "tax", "total"],
"additionalProperties": false
}

What OpenAI says about PDF inputs, the model and prices

PDF files: On models with vision capabilities, such as gpt-4o and later models, the API extracts both text and page images and sends both to the model.
OpenAI docs, File inputs, read
Token usage: PDF parsing includes both extracted text and page images in context, which can increase token usage. In the Responses API, set detail to auto (the default), low, or high to control the amount of visual detail for PDF page images. Before deploying at scale, review pricing and token implications.
OpenAI docs, File inputs, read
File size limits: A single request can include more than one file, but each file must be under 50 MB. The combined limit across all files in the request is 50 MB.
OpenAI docs, File inputs, read
The detail setting only affects PDF page image processing. Text extracted from the PDF is still included. Chat Completions file inputs don't support detail.
OpenAI docs, File inputs, read
GPT-5.4 Mini brings the strengths of GPT-5.4 to a faster, more efficient model designed for high-volume workloads. Input: Text, image. Output: Text. 400,000 context window. 128,000 max output tokens. Aug 31, 2025 knowledge cutoff. Snapshots: gpt-5.4-mini, gpt-5.4-mini-2026-03-17.
OpenAI docs, GPT-5.4 Mini, read
Model | Short context input | Short context cached input | Short context cache writes | Short context output. gpt-5.4-mini | $0.75 | $0.075 | - | $4.50
OpenAI API pricing, read
Structured Outputs is available in our latest large language models, starting with GPT-4o. For new projects, start with gpt-6-astra. Older models like gpt-4-turbo and earlier may use JSON mode instead.
OpenAI docs, Structured Outputs, read

The docs describe a PDF page as extracted text plus a page image, both sent to the model, and say that raises token usage. As of 2026-09-03 the pricing page has no line on how a PDF page is billed, so the per-page cost below is the measured figure.

Cost per 1,000 pages

gpt-5.4-mini as measured in the comparison beside Velrim, per 1,000 pages
Per 1,000 pagesgpt-5.4-mini directVelrim
List price for a PDF pageNo line on the pricing page
Measured in the comparison, prompt cached~$1.0 (measured, prompt-cached)
List price, grounding and confidence included$20

Velrim's published comparison of six extraction setups (September 2026) measured ~$1.0 (measured, prompt-cached) per 1,000 pages over 319 pages, with the repeated prompt prefix billed at the cached input rate of $0.075 per million tokens and the answer at $4.50 per million. The Gemini rows in the same table came to ~$2.2 (measured, token-priced).

What the comparison measured

Headline table from Velrim's September 2026 comparison: accuracy, fabrication on absent fields, error among the top 90% by confidence, and price per thousand pages, six setups
Armmacro-F1 (norm)fabrication (absent fields)wrong among the top 90% by confidence$/1k pages
Velrim0.69111.5% [6.1, 19.8]36.1% [29.9, 44.1]$20 (list; measured matches)
Gemini free-decode0.70917.0% [10.1, 26.7]not requested~$2.2 (measured, token-priced)
Gemini constrained0.69712.8% [7.6, 20.5]not requested~$2.2 (measured, token-priced)
gpt-5.4-mini free0.73510.8% [5.7, 18.4]none surfaced~$1.0 (measured, prompt-cached)
gpt-5.4-mini structured0.71810.4% [5.3, 17.6]none surfaced~$1.0 (measured, prompt-cached)
Mistral OCR 40.71240.3% [31.0, 52.5]none surfaced$5 (list; measured $5.3)

124 public documents, 2102 fields, 96 of them absent by the audited answer key. Free-decode gpt-5.4-mini invented a value for 10.8% of the absent fields, interval [5.7, 18.4]. With Structured Outputs, 10.4%, interval [5.3, 17.6]. Accuracy moved from 0.735 to 0.718 macro-F1, the highest two rows in the table, against Velrim's 0.691. Both rows completed 372/372 attempts.

Fabrication table from Velrim's September 2026 comparison: pooled rate, the all-attempted rule, the answer rate when the field is present, and completed attempts, six setups
Armfabrication (pooled, n=96)all-attempted ruleanswered when field presentcompleted/attempted
Velrim11.5% [6.1, 19.8]11.5% [6.1, 19.8]88.0% [84.0, 91.2]372/372
Gemini free-decode17.0% [10.1, 26.7]15.6% [9.3, 24.4]96.5% [95.0, 97.7]328/372
Gemini constrained12.8% [7.6, 20.5]11.1% [6.6, 18.1]96.7% [95.4, 97.9]315/372
gpt-5.4-mini free10.8% [5.7, 18.4]10.8%96.1% [94.6, 97.3]372/372
gpt-5.4-mini structured10.4% [5.3, 17.6]10.4%95.6% [93.0, 96.9]372/372
Mistral OCR 440.3% [31.0, 52.5]40.3%96.6% [95.2, 97.9]372/372
Planted trap fields from Velrim's September 2026 comparison: fields that cannot exist in that document type, and how often each setup answered them anyway
Armprobe fabricationn
Velrim20.7% [15.3, 27.2]198
Gemini free-decode18.0% [13.1, 23.5]183
Gemini constrained20.0% [14.6, 26.4]165
gpt-5.4-mini free37.9% [30.1, 46.3]198
gpt-5.4-mini structured39.9% [32.5, 48.4]198
Mistral OCR 453.5% [45.5, 61.5]198

The trap rows are the second fabrication measure: fields planted in the schema that cannot exist in that document type, so any answer is an invention. gpt-5.4-mini answered 37.9% [30.1, 46.3] of them free-decode and 39.9% [32.5, 48.4] with Structured Outputs, against 20.7% [15.3, 27.2] for Velrim. On the other side, the model answered 96.1% [94.6, 97.3] of the fields that were on the page and Velrim 88.0% [84.0, 91.2], so Velrim's lower rates are paid for in blanks.

The same call through Velrim

extract.ts
import { z } from "zod";
import { VelrimClient } from "@velrim/sdk";
const Invoice = z.object({ invoice_number: z.string(), total: z.number() });
const velrim = new VelrimClient({ apiKey: process.env.VELRIM_API_KEY! });
const result = await velrim.extract(Invoice, { bytes: pdfBytes });
result.data.total; // number, parsed with the schema
result.field("/total")?.state; // "present" | "null" | "missing"
result.field("/total")?.confidence; // 0 to 1
result.field("/total")?.anchor; // { page, bbox, snippet, page_dims }
one field of the response
"/total": {
"state": "present",
"value": 4210.55,
"confidence": 0.89,
"anchor": {
"page": 2,
"bbox": [451.0, 700.2, 512.4, 714.0],
"snippet": "Total due $4,210.55",
"page_dims": { "width": 612, "height": 792 }
}
}
gpt-5.4-mini called directly and Velrim, on what comes back and on the comparison's numbers
gpt-5.4-mini directVelrim
What comes backJSON in the shape of the schema. With response_format and strict: true, the shape is enforced as the model writes.The same JSON, plus a state, a 0 to 1 score and a page location for every field.
Fabrication on absent fieldsgpt-5.4-mini free: 10.8%. gpt-5.4-mini structured: 10.4%.11.5%
Planted trap fields answered37.9% [30.1, 46.3] and 39.9% [32.5, 48.4]20.7% [15.3, 27.2]
Accuracy, macro-F10.735 and 0.7180.691
Answered when the field is present96.1% [94.6, 97.3] and 95.6% [93.0, 96.9]88.0% [84.0, 91.2]
Same document, three runs, answers differ19% and 16% of repeats36% of repeats, nearly all on nested line-item tables
Per-field confidenceNone. logprobs was declined on a PDF file input.One 0 to 1 score per field, error curve published.
Price per 1,000 pages~$1.0 (measured, prompt-cached)$20

The score is one number per field, 0 to 1, with its error curve on the error curves page and the definition of the fabrication measure on the metric page. Grounding is a page number, a box and the matched text from the PDF's own word map, no second model call. Price is $0.02 per extracted page, prepaid from $5, about 500 pages free with no card.

Which to pick

gpt-5.4-mini with Structured Outputs, when accuracy per dollar is the whole requirement: it took the top two accuracy rows in the comparison at a twentieth of the price. Velrim, when a value needs a number to sort on and a place on the page to check it, and a field that is not on the page must come back empty, which the trap rows measure.