The copilot passed the demo. Nobody ran the exam.

There is no acceptance test, so there is no acceptance.

When Patronus AI built FinanceBench in 2023, the authors reported that GPT-4-Turbo with a retrieval system failed 81% of clear-cut financial questions over real SEC filings, questions a junior analyst would answer from the named document. The benchmark exists precisely because buyers had no way to test vendor claims. Model evaluation has since become the core of every emerging AI-assurance framework that a Big4 practice would sell against, and it is still routinely skipped at the moment it matters most: procurement.

The failure sequence is depressingly standard. The vendor demos on friendly questions. A pilot runs with no defined pass criteria. The tool rolls out. Months later an analyst notices a wrong EBITDA figure in a board pack. Trust collapses, the tool is quietly abandoned, and there is no record of what was ever tested. At no point did anyone define what "good enough" means for this team's numbers, or measure the tool against that definition. The gap is a discipline any accountant already knows from controls testing. Define the criteria, sample against them, keep the evidence.

Average accuracy is the wrong exam, and a model is the wrong examiner.

Two naive moves dominate current practice. The first is to score copilots on raw accuracy: the share of questions answered correctly. That metric rewards exactly the wrong behaviour, because a copilot that answers everything confidently will beat a careful one on coverage while planting wrong numbers in real workpapers. In a finance function the loss is asymmetric. An analyst who reads "I could not find this figure" loses a minute. An analyst who reads a confident, wrong EBITDA loses the board pack. A wrong answer costs more than no answer, and an evaluation that ignores that fact will select for confident guessers.

The second naive move is to let a stronger LLM grade the candidates. Now the evaluation inherits every weakness it was supposed to measure: the grader drifts between versions, extends partial credit to eloquent nonsense, and cannot be replayed to the same verdict twice. An acceptance test whose referee is itself unvalidated is not evidence. It is another demo.

The referee is code, the answers stay sealed, and the human prices the risk.

The bench draws the same line every system on this site draws, applied this time to evaluation itself. The golden set, 30 FinanceBench questions with reference answers and evidence excerpts, is held by referee code; candidates receive question text only. Scoring is fully deterministic: the referee parses the numeric answer and compares it within a published ±1% tolerance, checks any citation against the reference evidence by token overlap with a published 70% threshold, and books an abstention as its own outcome. No model ever grades a model. The one genuine judgement in the process, how much a wrong answer costs relative to a missed one, is not buried in a config file. It is handed to a named human as a slider, and the sign-off is written to the ledger with the multiplier attached.

Here is the deterministic core from the shipped file. The referee's verdict per question and the governed score the slider recomputes:

// the load-bearing decisions, no model anywhere in them.
function scoreItem(item, ans) {
  if (ans.answer == null) return { outcome: "abstain", ... };   // declining is an outcome, never an error
  const refVal  = parseNum(item.answer);                    // reference, held by the referee
  const candVal = parseNum(ans.answer);
  const outcome = numbersMatch(refVal, candVal) ? "correct" : "wrong";  // |Δ| ≤ max(1%·|ref|, 0.005)
  const ov = ans.cite ? overlapFrac(ans.cite, item.evidence) : null;   // token overlap vs evidence
  return { outcome, refVal, candVal, ov, citedOk: ov != null && ov >= 0.7 };
}

// the governed score: k is the human's risk appetite, wrong costs k× a miss
function governed(c, k) { return c.correct - c.abstain - k * c.wrong; }

// the headline metric is computed, never typed in: smallest k where the
// cite-or-abstain wrapper outranks the confident guesser · evaluates to 3
for (let k = 1; k <= 10; k++)
  if (governed(RES["WRAP-CA"].counts, k) > governed(RES["GUESS-1"].counts, k)) return k;

One slider, three candidates, and a leaderboard that changes its mind.

The three candidates are scripted personas, labelled on screen, built to make the arithmetic visible. The confident guesser answers all 30 questions and never offers a source; 19 of its answers are right. The ungoverned copilot answers 28 with loose citation discipline; three of its citations are invented, and the referee rejects each one by overlap. The cite-or-abstain wrapper may answer only when it can quote the reference evidence; it answers 11, gets 10 right, and declines the other 19 with "abstained: no evidence." Now drag the slider.

The score is correct − missed − k × wrong, so at 1× a wrong answer costs exactly what a miss costs and the board ranks on raw accuracy. The guesser's 19 hits put it first. By 3× its 11 wrong answers cost 33 points and the wrapper, carrying one wrong answer, moves ahead. It never comes back. Every line of that arithmetic is printed on the board next to each candidate, and the sign-off button writes the chosen multiplier and the resulting ranking into the ledger under the risk owner's identifier.

The numbers, their provenance, and the limits stated flatly.

The golden set is real. It is a 30-question extract of the 150-question FinanceBench open set: questions, reference answers, justifications and evidence excerpts over named SEC filings, fetched by a shipped script with the selection rule documented. The data carries a CC-BY-NC-4.0 licence, non-commercial, which suits a portfolio demo and is stated on screen; the harness itself is mine and accepts any golden set an engagement would substitute. The 81% failure figure is the FinanceBench paper's own headline measurement, quoted as the reason this bench should exist, never as this demo's result.

The candidate transcripts are not real model runs. Live multi-model scoring is impossible in a static browser page, so the three candidates are pre-computed, seeded transcripts, scored and re-ranked live by the referee, and the app labels each one as a scripted persona. The outcome counts were authored so the crossover arithmetic is visible at a realistic scale; the generator then re-runs the same referee logic and refuses to ship unless the scored outcomes and the 3× crossover actually emerge from the data. What the demo proves is the harness: sealed answers, deterministic scoring, priced abstention, replayable evidence. What it does not prove is any ranking of frontier models, and it says so.

• The extract covers numeric questions over US GAAP filings only. Free-text and multi-step reasoning answers need richer scoring than a ±1% tolerance compare.
• String-overlap citation checking is approximate. It catches invented quotes; it cannot certify that a genuine quote entails the answer.
• Scores here demonstrate the instrument. A production acceptance test would run live candidates on a held-out set sized with the audit team, with the same referee and the same ledger.

Set the price of being wrong, then sign your name to it.

Drag the appetite slider and watch the leaderboard reorder with the arithmetic printed line by line. Open any of the 30 questions to see the referee's worksheet: the sealed reference answer, the parsed values, the tolerance check, and each citation verified or rejected by overlap. Then sign off a ranking, replay every run from the stored transcripts, and tamper with a score to watch the chain break at exactly that row.

Sources. Islam et al., FinanceBench: A New Benchmark for Financial Question Answering, 2023 (Patronus AI): GPT-4-Turbo with retrieval failed 81% of clear-cut questions. Dataset: PatronusAI/financebench, CC-BY-NC-4.0, 150 open questions over SEC filings. Full provenance and the deterministic selection rule: DATA.md in the demo directory.