AI assurance · model evaluation · live

The acceptance test your finance copilot never took.

Finance teams are buying LLM copilots on vendor demos. On FinanceBench, the open benchmark built for exactly this problem, GPT-4-Turbo with retrieval failed 81% of clear-cut questions over real SEC filings. This bench runs the acceptance test the procurement step skipped: three candidate copilots answer a 30-question FinanceBench golden set, deterministic code parses and scores every answer (no model ever grades a model), abstention is scored as its own outcome, and you, the risk owner, set what a wrong answer costs. Every run lands in a hash-chained, replayable ledger, so the evaluation itself is auditable. Public dataset: FinanceBench · CC-BY-NC-4.0 · candidate transcripts scripted

golden questions · real filings

penalty where governance wins

wrapper cited or abstained

ledger replay verified

Risk-appetite console

you decide what a wrong answer costs · the ranking recomputes deterministically

1× cost of a wrong answer, as a multiple of a missed answer

at 1× a wrong answer costs the same as a miss, so the board ranks on raw accuracy

score = correct − missed − 1 × wrong · a wrong answer never beats an abstention

nothing is adopted until a named human signs · H-01 · risk owner

Golden set

30 FinanceBench questions · answers held by the referee

Candidates never see the reference answers or evidence. The referee holds both. Data © Patronus AI, CC-BY-NC-4.0, non-commercial; provenance in DATA.md.

Referee worksheet

select a question

AI Control Ledger, append-only, hash-chained ·

Each entry's hash covers its contents AND the previous hash (FNV-1a, computed in-browser). Replay re-derives every logged score from the stored transcripts; edit any record after the fact and the chain breaks at exactly that row. The run that justified a deployment decision stays reproducible.