Identity
When the deal starts and how big it is.
start_datecurrencyusage_plan_classcontract_valueMethodology
FinePrint's contract methodology measures whether a model reads billing terms correctly. Flexprice Model Intelligence provides a broader view of model capability, preference, and adoption.
Finance teams still turn contracts into data by hand. We built FinePrint because that work is too costly to get wrong.
Companies still turn contracts into structured data by hand. Poor contract management costs roughly 9% of revenue. High-stakes text in, reliable records out.
Under ASC 606 / IFRS 15, the contract drives recognized revenue. A misread fee or cadence becomes a billing error or audit finding. FinePrint measures who is ready.
We hand the model a real contract. It has to find every billing term we score.
A real PDF, read line by line.
01Every billing term, structured.
02Compared against our answer key.
03Ranked on accuracy, cost, and speed.
04These are the billing fields we expect on every scan. Miss one, and the invoice breaks.
When the deal starts and how big it is.
start_datecurrencyusage_plan_classcontract_valueThe core recurring charge for the product.
amountfrequencytimingInfrastructure or environment charges.
amountfrequencytimingMetered or consumption-based pricing.
amountfrequencytimingPrepaid credits or promotional balances.
amounttypeWhat the customer is allowed to use.
quantityunitperiodMinimum spend and true-up rules.
amountperiodoverage_factortrue_upCustom rates that break the standard schedule.
per-unit ratesotherWho the invoice goes to.
nameemailaddressNothing synthetic in our set. Every document was signed by real parties.
Executed commercial agreements under confidentiality, plus material-agreement exhibits from SEC EDGAR. No synthetic docs.
Order forms, MSAs, renewals, scanned pages, redlines, multiple currencies. The mix mirrors what a RevOps team sees, not the cleanest examples.
What makes them hard
Real contracts are messy. These eight patterns show up in almost every run.
Fees split across columns, not clean prose.
$240k over four quarters. Models must split it.
$10k/qtr and $40k/yr must both score.
Term, coverage, and billing get conflated.
Only the edited figure counts.
USD, EUR, GBP, INR from context.
Scanned PDFs with imperfect text.
MSA plus order form; order form wins.
We hold the answer key back on purpose to keep the board honest.
Ground truth is hand-checked and never published — we only share counts, not the data.
The leaderboard shows aggregates only, with contracts appearing as Doc A–F.
Filings from a long tail, rotated as the set grows.
One accuracy score is never enough. We publish six metrics so the trade-offs stay visible.
Fields the model got right, across every contract and run.
Of HIGH-confidence answers, how many were wrong.
Run-to-run spread. Low σ means repeatable.
Projected cost to read 1,000 contracts.
Median and tail wall-clock time per extraction.
Accuracy points per dollar spent.
We normalize both sides before we compare. Quarterly and annual fees can both score if the math matches.
Example
One contract, a few fields from a single extraction run.
| Field | Expected | Predicted | Result |
|---|---|---|---|
recurring_fee.amount | 25000 | 25000 | Match |
recurring_fee.frequency | quarterly | quarterly | Match |
usage_fee.amount | 180000 / yr | 45000 / qtr | Annualized |
recurring_fee.timing | advanced | n/a | Wrong |
fixed_fee.amount | 0 | 0 | Not scored |
scope_notes | “$250k capacity…” | “annual usage…” | Soft match |
Leaderboard accuracy is correct ÷ scored across all fields and 1 runs per model.
Every contract runs multiple times per model. Humans stay in the loop on every label.
Each contract runs 1× per model. We keep the full distribution.
Run-to-run σ sits next to accuracy on the board.
Ground truth is hand-labeled. Disagreements go to a human adjudicator.
Most of the stack is open. The contracts and labels stay private, by design.
The extraction harness, adapters, and field-level rubric are open source. Only the contract corpus and labels stay private.
Gold set is still small while the corpus grows, so σ matters. Extraction depends on OCR quality. Schema targets commercial billing terms, English-first today.
Data acknowledgement: Source: OpenRouter (openrouter.ai/rankings), as of 2026-10-04. Used under CC BY 4.0.
Flexprice Model Intelligence scores each distinct OpenRouter chat model on the same 0–100 scale. Moving aliases and routers stay in the playground picker because they do not identify one stable model.
Capability: We use the published Artificial Analysis Intelligence Index when a model has one. Otherwise, a fixed model-card score uses context length (25 points), maximum output length (15), image input (10), file input (5), tool use (10), structured output (10), and reasoning support (10). That 85-point feature score is scaled to 100 and calibrated against models with published Intelligence Index values using a linear fit with 15% slope shrinkage. Text-capable models receive a 10-point floor. This is an estimate of the model's feature envelope, not a measured intelligence result.
Preference: We log-scale the number of public OpenRouter requests per model. When an Arena rating exists, preference is 70% of its normalized rating plus 30% of the request score; otherwise it is the request score. This combines developer selection with direct human comparisons where available. It does not claim that request volume equals an Arena vote.
Adoption: We log-scale public OpenRouter token volume over its monthly window. A model absent from the published activity feed receives zero for request activity and adoption until it appears there.
Index: Models with Flexprice task results: 30% capability + 25% preference + 20% adoption + 25% Flexprice tasks. Models without task results: 40% capability + 35% preference + 25% adoption. Scores are relative to the comparison set, so they can move when the catalog, published activity, or available task results change. The Index is a composite, not a standalone contract-extraction accuracy score.
Flexprice tasks: The local Flexprice task composite is 0.35 × accuracy + 0.30 × extraction + 0.20 × convention + 0.15 × max(0, 100 − hallucination) − 0.20 × consistency, clamped to 0–100 and rounded to one decimal. With two or more matched models, composites are min-max normalized to 0–100; if all composites are equal, they normalize to 100. A single matched model keeps its composite score. The Index uses this normalized task score. These results are from our local tasks benchmarking.