WTWilly Tai
Case studyAI Governance & Assurance

The internal audit, run end to end.

A governed agent re-derives every procurement exception from the raw records. It works over the whole ledger rather than a sample, cites each exception to the invoice it came from, and never touches a number with a language model. This walks through exactly how it works, where the hard parts are, and what it would take to break it.

01 / The pain

You test a sample, and the exceptions live in the part you didn't pull.

An internal auditor reviewing procurement can't read every transaction, so they pull a sample, maybe 25 of a few hundred invoices, and match each by hand against its purchase order and goods-receipt note. It takes weeks, it's point-in-time, and the duplicate payment or the overbilling that matters is, by definition, just as likely to sit in the 90% nobody looked at. When teams reach for AI here, they usually do the one thing you should never do: they point a language model at the ledger and trust the number it returns.

02 / Why the naive approach fails

"Just check the invoice equals the PO" breaks on contact with real data.

The obvious version of three-way match is a single equality: invoice.amount == po.amount. It would pass a demo and fall apart in a real accounts-payable ledger, because procurement is messy in specific, knowable ways:

So the real job isn't arithmetic. It is a join across three tables with the right keys, the right tolerances, and the right cross-record comparisons, plus an honest decision about where a rule ends and a human's judgment begins.

NAIVE invoice.amount == po.amount ✗ partial delivery✗ price tolerance✗ duplicate (cross-invoice)✗ qty over receipt✗ segregation of duties REFINED · three-way join + tolerances + cross-record PO Goods receipt Invoice join on po_id · tol ±1%+ window & approver checks 6 typed exceptions, each cited
fig.1 · why equality fails, and the join that replaces it
03 / How it works

The finding is computed in code. The model only writes the email.

This is the load-bearing decision of the whole system, so it's worth drawing precisely. Every number and every flag is computed in Python from the joined records, reproducible to the cent, re-runnable, and auditable. The language model sits downstream and read-only. It drafts the request-for-information email about a finding the engine already produced, and it must cite that finding's record IDs or it writes nothing. A model never decides whether an exception exists.

DETERMINISTIC · computed, reproducible MODEL · language only, read-only, cited IngestPO·GRN·Invoice Match enginePython · the rules Exceptionstyped · cited SixSigma RFI draftLLM · cite-or-abstain Human gateapproves scope Append-only, hash-chained ledgerevery action above is written here · tamper-evident · replayable
fig.2 · the trust boundary, where code ends and the model begins

Inside the match engine, each invoice is joined to its PO and goods-receipt note on po_id and run through five checks. The parameters are explicit rather than guesswork:

Invoice po_id · qtyunit_priceapprover · date join po_id unit_price > PO × 1.01price variance inv.qty > po.qtyqty over PO no GRN | inv.qty > grn.receivedno receipt / over-received po.approver == inv.approversegregation of duties PO + GRN qty · unit_priceapproverreceived qty + duplicate scan: same vendor & amount ≤ 7 days apart
fig.3 · the hard part, five parameterised checks over the three-way join
04 / One worked example

One real record, traced, and one the engine correctly leaves alone.

Take invoice INV-00001. The engine joins it to PO-00001 and GRN-00001, finds the invoiced quantity (32) exceeds the ordered quantity (28), prices the gap at the invoice unit price, and emits a typed, cited exception. Only then does the model draft the email, quoting those exact IDs.

InvoiceCheck that firedObservationValue at riskEvidenceResult
INV-00001Quantity over POinvoiced 32 units vs 28 orderedS$3,612.16INV-00001 · PO-00001 · GRN-00001flagged
INV-00088Price varianceinvoiced @ 0.6% over PO, inside the ±1% bandnoneINV-00088 · PO-00088correctly cleared
INV-90001Duplicate scansame vendor & amount as INV-00011, but 33 days apartnoneINV-90001 · INV-00011not a duplicate
Telling these apart is the whole job. A naive checker either flags every price difference, which is noise, or none of them, which misses real problems. This one clears the 0.6% variance because it sits inside the tolerance band, and it declines to call INV-90001 a duplicate because the 7-day window isn't met. Then it still catches the real quantity overbill. Getting the near-misses right is what separates a control from an alarm.
05 / Numbers & honest trade-offs

What it found, and what those numbers do and don't prove.

161
invoices tested
The full population, not a sample.
46
exceptions raised
Across 5 checks · 805 control opportunities.
S$481k
value at risk
Σ of each exception's priced gap.
~3.1σ
process quality
54,762 DPMO, illustrative.

Where these come from, honestly. Origin Company is a synthetic enterprise I generate, with exceptions seeded deliberately. So these figures prove the mechanism works, not that it survives a specific real ledger. The sigma figure is a process-quality index over five opportunities per invoice (805 total). It is illustrative on synthetic data, not a benchmarked claim.

The trade-off every auditor actually buys on is noise. 46 exceptions on 161 invoices is a deliberately high seed rate. In production the lever is the tolerance band. Widen the price tolerance from 1% to a contracted 2.5% and the price-variance count falls sharply, trading recall for fewer false positives. The engine doesn't pretend a flag equals a problem. It produces a cited candidate list a human triages, and the band is the dial that controls how much lands on their desk. A continuous version (see a continuous-monitoring version of this engine) adds the other half: only what's new since the last run reaches the queue, so the daily volume is the delta rather than the whole book.

06 / Governance, shown

The append-only log is real. Here is a row of it.

Every action, from the scope approval to each finding to each RFI, is written to a hash-chained ledger. Each entry's hash covers its contents and the previous hash, so altering any record after the fact breaks the chain at exactly that row. The decision fields are the gate's structured output. The model never writes them.

AI Control Ledger · entryintegrity verified · replay 46/46
seq        : 17
agent      : Finance Audit Bot
action     : raise_rfi
subject    : INV-00001
tier       : low          decision : escalate → control owner
evidence   : [INV-00001, PO-00001, GRN-00001]
prev_hash  : 9f2c…a17b
hash       : d4e8…c012   ← = sha256(prev_hash + canonical(entry))
Threat model: the ledger is append-only. A flipped decision or amount changes the hash, which breaks the chain at the altered seq. The tamper test runs live on the home page.
07 / Old vs new

The same audit, re-engineered.

TRADITIONAL · ≈ 2–3 weeks pull sample 25/161 match by hand 136 invoices never seen RE-ENGINEERED · minutes all 161 · deterministic 46 exceptions, each cited human approves scope + RFIs
fig.4 · weeks on a sample become minutes on the whole book

What this says about how I work.

I don't point an AI at a ledger and trust the output. I draw the trust boundary first, with the calculations in code and reproducible to the cent, and the model strictly downstream and cited. Then I confront the hard parts out loud: the edge cases that defeat the naive version, the tolerance that controls the noise, the near-miss the system must get right. Anyone can sample twenty-five invoices. The harder and more useful thing is to run the control over all of them, name the tolerance that separates a real finding from noise, and keep the model's hands off every number it touches. That discipline is what makes the output worth acting on, not the model.

See it run.

The artifact above is real output from the engine over Origin's ledger. The full interactive console, with its risk-ranked audit universe, the 46 exceptions, and the RFI drafts, runs as a working application.

See the live governance demo →The full console runs on my machine. Ask me for a 90-second walkthrough or a screen-share.