WTWilly Tai
Case study · AI governance & assuranceLive demoPublic SEC data

Completeness testing that shows its evidence, sentence by sentence.

Every audit senior has lived this task: the IFRS disclosure checklist, run against a draft annual report in the final days before signing, with Ctrl+F and stamina as the method. Disclosure Sentinel runs a 60-requirement checklist against SAP SE's real Form 20-F from SEC EDGAR. For each requirement it either shows the exact sentence in the notes, verified by code before it reaches you, or it says not found and makes a named human decide what that gap means. Coverage is a number the engine computes from verified citations, and the entire run replays from a hash-chained ledger.

60
requirements, rule-tailored
applicability switched by the entity profile, in code
58/60
located with code-verified quotes
on the seeded-gap copy of the real filing
2
abstentions, both planted
each routed to a named human disposition
0
unverifiable citations shown
the verifier rejects anything not literally in the text

SAP SE Form 20-F, FY2025 · SEC EDGAR accession 0001104659-26-020058, public domain · checklist paraphrased from IAS 1, IAS 7, IFRS 7, IFRS 15, IFRS 16

01 · The 168-page endurance event

The last control before signing is a tired person with a PDF.

Disclosure checklist completion is the least glamorous work in an audit and among the most consequential. A senior downloads the firm's checklist, tailors applicability by ticking entity characteristics, then works through hundreds of yes, no and N.A. questions against a draft financial statements PDF that can run past 300 pages. Cross-references are traced by hand. Sign-off happens late, a reviewer spot-checks a sample, and the liability sits with the signing partner.

The exposure is well documented. Disclosure deficiencies recur in PCAOB inspection reports and in the FRC's Annual Review of Corporate Reporting year after year, and major-firm IFRS disclosure checklists run well past 150 pages. When a missed disclosure surfaces in cold review or regulator inspection a year later, there is usually no record of what was actually checked against which text. The control, in practice, is human stamina.

02 · Why a chatbot cannot sign this

Paste the PDF into a model and you get confidence without evidence.

The naive automation is to hand the draft accounts to a large language model and ask "is IFRS 15.120 disclosed?" The failure modes are exactly the ones an auditor cannot accept. The model paraphrases where the checklist needs verbatim text. It fabricates plausible note references. Worst of all, it treats silence generously: asked about a disclosure that is absent, a helpful model tends to find something that sounds close enough. Completeness testing is the one task where that instinct is disqualifying, because the entire point is to notice what is missing.

There is also a subtler problem: applicability. Whether the IFRS 7 section applies is a matter of explicit rules over entity facts, and letting a model guess it means the scope of the whole exercise rests on an inference nobody can inspect. So this demo splits the work the way the trust boundary demands. Rules decide scope. The model only points at sentences. Code checks the pointing. A person owns every gap.

03 · The trust boundary, wired in

Five steps, and the model is only trusted with one of them.

Step 1, applicability. An entity-profile form (listed, financial instruments, revenue from contracts, leases, business combination this year) switches checklist sections on and off through explicit rules, each shown beside the section it toggled. SAP's profile turns all five sections on: 60 applicable requirements. Untick "business combination this year" and the two IAS 7.40 acquisition items drop out, visibly and by rule. Step 2, locate and cite. For each applicable requirement, the locator must return a passage from the extracted filing or return not-found. In this demo the locator is a deterministic stub, and the page says so; the architecture is what a real model would be bolted into. Step 3, verification. A code-level checker rejects any citation whose text is not literally present in the document. A failed check displays as a failure. Step 4, the human gate. Every not-found routes to a person who dispositions it from a controlled list (disclose, immaterial, not applicable), with name and timestamp. Step 5, the ledger. Coverage is computed by code from verified citations only, and the full run (profile, rules fired, counts, digest, dispositions) is hash-chained and replayable.

DISCLOSURE SENTINEL · ONE REQUIREMENT, INSTRUMENTED Entity profile 5 questions → explicit rules deterministic scope: 60 items MODEL (STUB) · LOCATE ONLY verbatim passage or not-found · nothing else DETERMINISTIC VERIFIER quote literally in text? yes → verified citation no → shown as failure coverage counts verified only Human gate every not-found Hash-chained, replayable ledger every run: profile, rules fired, counts, digest · every disposition: decision, name, timestamp document under test: SAP SE Form 20-F extract (SEC EDGAR, public domain) · a second copy with two disclosures removed provides the seeded gaps
One requirement, instrumented: rules set the scope, the model points, code checks the pointing, a person owns the gaps, and the ledger remembers all of it.

Here is the load-bearing decision code from the shipped app: the locate-verify-count core. Coverage can only rise when a quote survives the literal-presence check, so an invented citation cannot move the number.

// Step 2 · the locate-and-cite stub: first passage containing every cue, or null.
function locate(item, passages) {
  const cues = item.cues.map(c => c.toLowerCase());
  for (const p of passages) {
    if (cues.every(c => p.text.toLowerCase().includes(c))) return p;
  }
  return null;   // silence is an abstention, routed to a human · never a pass
}

// Step 3 · deterministic verification: the quote must be literally present.
function verifyQuote(quote, docText) { return docText.indexOf(quote); }

// Coverage is computed from verified citations only.
const off = verifyQuote(p.text, docText);
status = off >= 0 ? "verified" : "unverifiable";
04 · Worked example

The same requirement, on two copies of the same filing.

Take D-05, the classic: IFRS 15.120, transaction price allocated to remaining performance obligations. On the filed extract, clicking the row jumps the viewer to the sentence in Note (A.1) that answers it, beginning "The transaction price allocated to performance obligations that were unsatisfied or partially unsatisfied as at December 31, 2025, was €90.5 billion", with the verifier's badge showing the character offset where the quote sits in the document text. That is a tick that carries its evidence.

Why the gaps are planted: a completeness demo on a complete filing would only ever show green. SAP's disclosures are, unsurprisingly, complete for these 60 requirements. The two removed passages let you experience the abstention and the human gate on a document that is otherwise real from first character to last.
05 · What the numbers mean, and what they do not

Real filing, real checks, honest boundaries.

The document is real: SAP SE's Form 20-F for fiscal year 2025, fetched from SEC EDGAR (public domain) and curated to a 60-passage extract by a stdlib-only script that validates every passage against the source and aborts on any drift. Narrative passages are verbatim; three are linearized rows of the filing's HTML tables and the app labels them as such. The metrics are genuinely computed: 60 of 60 requirements locate and verify on the filed extract, 58 of 60 on the seeded-gap copy with exactly the two planted abstentions, and zero unverifiable citations reach the reviewer in either run.

The limits are just as real. Sixty requirements are a teaching subset, an order of magnitude short of a firm checklist, and my paraphrases are illustrative rather than authoritative IFRS text, which is IFRS Foundation copyright. One company, one framework, one period. The locator is a deterministic stub, so the verifier that would earn its keep against a probabilistic model never fires here, and the page says so plainly. PDF-to-text extraction quality bounds recall on heavily tabular notes in any production version of this. And a green checklist is evidence support for a human conclusion about disclosure adequacy. It is never the conclusion.

06 · See it run

Click a requirement. Then break the ledger.

Tailor the profile and watch sections switch by rule, jump from any row to the exact sentence in SAP's notes, flip to the seeded-gap copy and disposition a gap under your own name, then tamper with the ledger and watch Verify locate the edit.

▶ Open the live demo Runs entirely in your browser · public SEC EDGAR data · deterministic, replayable engine

The checklist as a control, at last.

I have signed off disclosure checklists at 11pm, and I have seen what a cold review does to a tick that cannot show its evidence. The instinct from that career is exactly the architecture here: never accept an assertion you cannot re-perform. What engineering adds is the means to enforce it mechanically, a locator that is only allowed to point, a verifier that rejects anything not literally in the text, and a ledger that makes the whole run repeatable a year later, when the inspector asks. If your firm is putting AI anywhere near financial reporting assurance, this is the shape I would insist on before the first pilot touches a client file.