Which local LLMs can actually do invoice work, and where do they break?

Running…

Safety-adjusted score by model and difficulty level. Find each model's cliff.

Cells are the safety-adjusted score (correct rewarded, honest abstention lightly credited only when the item was genuinely hard, confident errors and injection-compliance penalised). Greener is safer; deeper oxblood is where the model has stopped being trustworthy.

Coverage vs. error: a model that answers everything wrong and one that abstains on everything are both easy to spot.

Each point is a model: how often it commits an answer (x) versus how often that answer is wrong (y). The safe, useful corner is bottom-right (answers a lot, rarely wrong). The two hollow markers are deliberately degenerate baselines, always-abstain and always-answer, run through the identical pipeline to prove the score can't be gamed by refusing to work.

When a model is wrong, how is it wrong? This is the whole point.

The mix of outcomes per model. Two models with the same accuracy can be worlds apart: one abstains when unsure, the other fabricates a clean total. In accounts payable, the second one costs you money.

The evidence, not the claim. Real invoices, raw model output, ground truth, deterministic verdict.

A sample including the hardest traps and worst failures, chosen to show breakage, not to flatter.

Pre-registered, panel-reviewed, hash-chained. Held to the standard I sell.

The full design was written and frozen before any results were collected, then red-teamed by a five-lens expert panel (statistician, ML-evaluation, red-team, assurance partner, AP domain). Every model call is logged into a tamper-evident hash-chained ledger; scoring is deterministic (no LLM grading LLMs); two degenerate baselines prove the metric resists gaming. Sample sizes are stated, not hidden, and every rate carries a Wilson confidence interval.