Methodology & transparency
How FinePrint measures a model.
FinePrint scores one real, economically-important task: reading a messy commercial contract and turning it into correct, structured billing data. This page documents exactly what the dataset is, the complexities it contains, the metrics we report, how we score, and how the benchmark is kept rigorous and hard to game — so anyone can judge the results on their merits.
Why this task matters
Every company runs on contracts, and turning them into structured data is still overwhelmingly manual — slow, expensive, and error-prone at volume. World Commerce & Contracting estimates poor contract management costs organizations roughly 9% of annual revenue, with a single basic contract taking ~$6,900 to process. It is the canonical document-AI problem: unstructured, high-stakes text in, reliable structured records out.
Contract-to-cash lives or dies on these exact fields. Under ASC 606 / IFRS 15, the contract itself is the legal basis for recognized revenue — so a misread fee, cadence, or commitment becomes a mis-invoice, a revenue-recognition error, or an audit finding. As usage-based and hybrid pricing spread (~45% of SaaS companies now bill on some usage model), the terms get more intricate. This is the task Flexprice automates in production; FinePrint measures who’s ready.
The task
A model receives a real contract as numbered OCR lines plus a structured markdown view, and must return a fixed billing schema as strict JSON — citing the exact source line for every field. It is the Flexprice production task, not an academic exam.
The billing schema
~13 hard-scored fields · free-text notes reviewed, not string-matchedThe dataset
Real, license-clear public contracts — material-agreement exhibits from SEC EDGAR filings (public records) and the CC-BY-licensed CUAD corpus of commercial agreements. No synthetic documents: every contract was signed by real parties.
Order forms, master service agreements, renewals and amendments, across industries and currencies — including scanned and redlined documents. The mix is chosen to mirror what an accounts-receivable or RevOps team actually sees, not the cleanest examples.
Where the difficulty comes from
real-world complexity, not trick questionsResults shown here run on a hand-labeled seed. The corpus is expanding toward ~1,000 contracts via a two-tier scheme: a gold human-labeled holdout for scoring, plus a larger silver pool (strong-model drafts, human-adjudicated) for coverage — the standard way credible benchmarks scale labels without sacrificing trust.
Kept honest & un-gameable
Ground-truth answers are hand-checked and never published — so no model can be trained on them. We disclose the volume, never the data.
Only per-model aggregates ship. Contracts appear as “Doc A–F”; no identities, values, or spans leak from the leaderboard.
Documents are drawn from a long tail of filings and rotated as the set grows, limiting overlap with any pre-training corpus.
The metrics
A single accuracy number hides how a model fails. We report a small, defined set — each with an exact definition — so the trade-offs are visible.
Share of scored fields the model gets right, aggregated across every contract and run — economic-equivalence aware.
Of the answers a model marked HIGH confidence, the share that were wrong. The metric that matters for a human-review pipeline.
Run-to-run standard deviation of accuracy. Low σ = a trustworthy, repeatable result; high σ = a coin flip on hard contracts.
Measured input/output tokens times published per-token pricing, projected to 1,000 contracts.
Median and tail request time — the difference between a real-time and a batch workflow.
Accuracy points bought per dollar. Surfaces the cheap-and-good models the headline rank hides.
How we score, field by field
Every field is normalized, then compared to ground truth. Both sides get the same treatment: numbers pulled out of strings like “$0.05/min”, dates canonicalized, a missing value treated as $0 where that’s equivalent. A field counts only if truth or prediction is non-empty. Fee amounts are matched on an annualized basis, so a correct answer in a different cadence still scores.
Worked example
one contract, a few fields| Field | Expected | Predicted | Result |
|---|---|---|---|
| platform_fee.amount | 25000 | 25000 | ✓ |
| platform_fee.frequency | quarterly | quarterly | ✓ |
| usage_fee.amount | 180000 / yr | 45000 / qtr | ✓ annualized |
| platform_fee.timing | advanced | — | ✗ wrong |
| hosting_fee.amount | 0 | 0 | · not scored |
| scope_notes | “$250k capacity…” | “annual usage…” | · soft (reviewed) |
Every scored field on every contract is recorded this way; the leaderboard number is Σcorrect ÷ Σscored over all of them, across 3 runs.
Statistical rigor
Every contract is run 3× per model. We keep the full distribution, not just a mean.
We publish run-to-run σ alongside accuracy — a single run can swing several points and mislead.
Ground truth is hand-labeled and adjudicated; as the set scales, agreement is measured across annotators.
Where FinePrint sits
FinePrint borrows the principles the field has converged on for credible evaluation, applied to a vertical the general benchmarks skip.
Not trivia — an actual job (resolve a GitHub issue; here, turn a contract into billing data).
Accuracy is necessary but not sufficient; we report cost, latency, calibration and consistency together.
Labels are held back so results resist training-set contamination and gaming.
A vertical the general benchmarks don’t cover: commercial contract → billing terms, for finance & ops.
Openness & limitations
The harness, the model adapters, the scorer and the aggregation are open source and unit-tested; anyone can run FinePrint on their own labeled data. Pricing is pulled from OpenRouter. Only the contract corpus and its labels are private — that’s the part that keeps the benchmark honest.
The scored seed is small while we scale to ~1,000; results carry run-to-run variance we report as σ. Extraction sits on an OCR step, so document quality matters. The schema targets commercial billing terms and is English-first today. We publish these limits rather than paper over them.