Overview
The dataset and schema-driven scorer are useful infrastructure, but current model reliability on enterprise schemas is low, so production use requires decomposition, extra validation, or postprocessing.
Citations0
Evidence Strength0.90
Confidence0.90
Risk Signals12
Trust Signals
Findings with numeric evidence: 5/5
Findings with evidence refs: 5/5
Results with explicit delta: 1/6
Reproducibility
Status: Code + data available
Open source: Yes
License: MIT
At A Glance
Cost impact: 60%
Production readiness: 25%
Novelty: 60%
Why It Matters For Business
Current LLMs can produce parseable JSON but are unreliable on large enterprise schemas; field-level correctness is the practical bottleneck for automating PDF→JSON pipelines.
Who Should Care
Summary TLDR
ExtractBench is an open-source benchmark and evaluation framework for converting PDFs into structured JSON under realistic enterprise schemas. It provides 35 (PDF, JSON Schema, gold JSON) examples across five domains and 12,867 evaluatable fields, and a schema-driven scorer that assigns per-field metrics (exact ID match, numeric tolerance, semantic text) and LLM-based array alignment. Baselines on current frontier models show fragile end-to-end reliability: 51% produced valid JSON but only a 4.6% overall pass rate; on a 369-field SEC schema all models produced 0% valid output. The repo and evaluation code are released under MIT.
Problem Statement
Real enterprise extraction needs reliable, schema-conformant JSON from PDFs. Existing benchmarks either test small schemas, text-only inputs, or only JSON validity. There is no public benchmark that (1) evaluates PDF→JSON at enterprise schema breadth and (2) provides per-field, semantics-aware scoring to distinguish omission, hallucination, and type-specific correctness.
Main Contribution
ExtractBench dataset: 35 PDF/JSON Schema/gold JSON triplets across 5 domains, 12,867 evaluatable fields.
Schema-driven evaluation: per-field metrics, three-state missing/null handling, and LLM-based array alignment.
Key Findings
Frontier models fail at enterprise-scale schemas.
End-to-end validity is not the same as correctness.
Results
| Metric | Value | Baseline | Delta | Split / Dataset | Evidence | Evidence Ref |
|---|---|---|---|---|---|---|
| Valid JSON rate (aggregate) | 51% (107/210) | — | — | All domains, 6 models | Fraction of extractions producing parseable, schema-conforming JSON | Table 5 |
| Aggregate pass rate (field-level, end-to-end) | 4.6% (844/18,516) | — | — | All domains, 6 models | Field scores use per-field metrics; invalid outputs count as zero | Table 5 |
What To Try In 7 Days
Run ExtractBench on your target schema to measure end-to-end pass rate, not just JSON validity.
Estimate output token volume (including expanded arrays) and test model capacity on a representative document.
Break large schemas into smaller chunks or decompose arrays to reduce output volume per call.
Reproducibility
Risks & Boundaries
Limitations
Only English documents.
Five domains only; not a broad-coverage corpus.
When Not To Use
For assessing multilingual extraction performance.
As a large-scale training corpus or to draw fine-grained statistical conclusions.
Failure Modes
Formatting errors: trailing commas and truncated JSON break strict parsers.
Omission vs hallucination: missing fields vs invented values require different handling.