Whether AI is governable is decided in the architecture, long before any policy document is written.
Willy Tai · governed AI for regulated operations
Every few weeks, a finance team somewhere discovers that its new AI assistant has been confidently wrong about a number, and the post-mortem reaches the same conclusion. The firm did have governance, but only as a document, and that document was never in the path of the decision.
I worked as an auditor before I wrote production code, and auditing leaves you with one durable instinct: you test controls, not intentions. A policy that says "AI must be explainable, fair and accountable" is an intention. It describes what the organisation would like to be true. On its own it does nothing to make any particular answer explainable, because it sits beside the system rather than inside it. When the regulator asks how you know the model didn't invent that figure, "we have an AI ethics policy" tells them nothing about that figure at all.
What actually governs an AI system is its architecture, and specifically one line drawn through it. I call that line the trust boundary. On one side is everything the model is permitted to decide and compute. On the other is everything done by deterministic code you can read and re-run. Where you put that line is the most consequential decision in a regulated AI build, and it is an engineering choice long before it is a committee one.
In every system I build, the boundary follows three rules. None of them is a guideline. Each is enforced in code, and each can be demonstrated to someone who doesn't trust me.
A language model is a magnificent instrument for turning structure into prose and prose into structure. It is a poor calculator and a worse ledger. So in an audit system, the model does not decide whether an invoice is an exception. Python does, by joining the purchase order, the goods receipt and the invoice and applying the tolerance the business signed off on. The model's job begins only after the finding exists: it drafts the email about it. Move that line by one step, so that the model gets to "review the totals," and you have built a system whose core number is a guess wearing a suit.
The real cure for a hallucination is not a truer statement but a willingness to abstain. A grounded system answers with the clause, the record, the transaction it is pointing at, and when nothing supports an answer it declines to give one. This is unnatural for a model, which is trained to be helpful and will fill a silence with something plausible. So the abstention has to be built around it as a hard gate: no citation, no output. That gate has an uncomfortable consequence, because the system will sometimes say "I don't know" to a senior person who wanted a number. I treat that as working as intended. In a regulated process, a confident wrong answer is far more expensive than an honest blank.
Automation earns its keep on the thousands of routine decisions. It loses a licence on the one that should have had a name against it. So the boundary includes a second kind of gate. Actions are risk-classified, and the consequential ones, such as releasing a payment, issuing a close, or referring a customer, escalate to a person who approves them on the record. The valuable agent in a regulated firm is frequently the one that refuses to finish: it reaches the end of the workflow, sees an open control, and stops. A system that knows when not to act is a more sophisticated thing than one that acts quickly.
Draw the boundary first and an interesting thing happens to the word "governance". It stops being a property you assert and becomes a property you can show. You don't tell the risk committee the system is auditable; you replay its log in front of them and let the chain break on a record you tampered with. You don't promise it won't overstep; you point at the gate and the escalations it produced last week. The controls no longer sit in a PDF in a different building from the code. They live in the code itself.
The policy document describes the car you would like to have built. The trust boundary is whether it has brakes.
This is also why "add an approval workflow later" is the wrong instinct. The boundary is not a feature you bolt on once the demo impresses people. It is the load-bearing wall. Decide it last and you have a system that already trusts the model with the numbers, and no amount of process wrapped around the outside can claw that trust back. Decide it first and everything downstream, the eval, the audit log, the human gate, has somewhere to attach.
None of this makes the AI less capable. It makes it boring in exactly the places a regulated business needs boring: the arithmetic is dull and reproducible, the citations are literal, the consequential decisions are slow and human. The interesting, generative, occasionally brilliant part of the model is still there. It just doesn't get to touch the parts that, when they go wrong, end up in a regulator's letter. That trade is the whole job. The teams that manage to ship AI into regulated operations are rarely the ones with the best models. They are the ones who drew the line in the right place and can show you where it is.
Written by Willy Tai. The patterns here are demonstrated across the case studies: the audit bot, the governed close, the conversational-analytics canvas. · More writing