WTWilly Tai
Case studyData & Knowledge

The answer is cited, or there isn't one.

Every enterprise fears the same thing about an AI assistant: that it will confidently make something up. This one can't. It answers policy questions with the cited clause itself, never prose generated on top, and when nothing scores high enough, it abstains. That refusal is built into the control flow, not bolted on as a disclaimer. Below is the score that decides, and where the approach reaches its ceiling.

01 / The pain

"Where's the policy on that?" And the assistant invents one.

People lose hours hunting for the right internal rule, searching a document store, pinging compliance, waiting. So teams reach for a chatbot and hit the worst failure mode in regulated work: a fluent, confident, wrong answer with no source. In a bank or a payments business, an answer you can't trace to a policy is worse than no answer, because someone will act on it.

02 / Why the naive approach fails

Keyword search misses; "generate from context" hallucinates.

So the design does three things: match by concept rather than just keyword, return the source clause itself so there is no generated text to hallucinate, and abstain below a grounding threshold. The model never writes the answer. It only points at one.

NAIVE retrieve → LLM generates ✗ keyword misses "interpretability"✗ generation invents to fill gaps✗ never says "I don't know" REFINED · concept score, extract or abstain score concept overlap return cited clause abstain ≥ threshold< threshold
fig.1 · generate-on-context vs concept-score, then extract or abstain
03 / How it works

Retrieval is deterministic. The answer is the source.

There is no generation step, so there is no surface for a hallucination. The model isn't in the answer path at all: retrieval scores each policy by concept overlap, and the assistant returns the winning clause verbatim with its ID and a grounding score, or, below threshold, it abstains.

DETERMINISTIC · score, threshold, extract NO MODEL IN THE ANSWER PATH Questionnatural language Score overlapterms + concepts ≥ 2.0? Cited clause + grounding scorethe policy text itself · POL-ID Abstainno clause clears the bar · it won't guess yesno
fig.2 · where the line sits. The model never writes the answer; only the library does.

The grounding score is a defined number, not a vibe. Query terms are expanded with a small concept map (so "interpretability" reaches "explainability"), then:

score = |q ∩ policy_terms| + 1.5 × |q ∩ title_terms| (q = concept-expanded query) if score < 2.0 → abstain grounding% shown = min(97, 50 + 6 × score)
fig.3 · the hard part. The grounding score and the abstain threshold.
04 / One worked example

A concept hit, a paraphrase hit, and an honest abstention.

QuestionMatchGroundingAnswer
"Can one person both raise and approve a payment?"POL-FIN-0297%returns the segregation-of-duties clause, verbatim, cited
"Is customer e-money kept safe?" (paraphrase)POL-SAFE-0197%concept map reaches "safeguarded" → the float clause
"What's our refund policy for a damaged laptop?"· nonebelow 2.0abstains; needs a human or a new policy
The abstention is doing the real work here. There is no refund policy in the library, so instead of stitching together a plausible-sounding answer, the assistant returns nothing and says why. A clean "I don't know" is exactly what makes it safe to put in front of staff in a regulated business.

05 / Numbers & honest trade-offs

The dial, and the ceiling.

10
policies indexed
The only source of truth.
0
generated tokens
Answers are extracted, not written.
2.0
abstain threshold
The precision/recall dial.
1
honest abstention
Shown, not hidden.

The trade-off is the threshold. Raise it and the assistant abstains more often, giving fewer wrong answers but more "ask a human". Lower it and it answers more, risking a weak match. In regulated work you set it high on purpose: a false abstention costs a few minutes; a false answer costs a breach. The honest ceiling: because the answer is the extracted clause, this can't synthesise across two policies. A question that needs combining the AML and the safeguarding rules will return the best single clause or abstain, rather than a merged answer. That is a deliberate floor under safety. A generative layer could synthesise, but only by reintroducing the hallucination risk this design exists to remove. For a compliance tool I would take the ceiling over the risk, and say so.

06 / Governance, shown

Cite-or-abstain is the contract.

Every response takes one of exactly two shapes. There is no third "confident guess" path. This is the safeguard, shown as the actual output:

Grounding contractgrounded
Q: "Can one person both raise and approve a payment?"
A: "No single individual shall both raise and approve a payment."
   citation : POL-FIN-02 · Segregation of duties
   grounding: 97%        basis: term + concept overlap, score ≥ 2.0
, , ,
Q: "What's our refund policy for a damaged laptop?"
A: ABSTAIN , no policy scores ≥ 2.0. Routed to a human.
No answer is ever produced without a citation. The only alternative to a cited clause is an explicit abstention.
07 / Old vs new

From a document hunt to a cited answer, or an honest stop.

BEFORE search · ping · wait or a confident, sourceless answer AFTER cited clause in seconds · or an honest abstain
fig.4 · from hunt-and-hope to a cited answer, or a clean refusal

What this says about how I work.

I treat the failure mode as the design centre. Everyone wants the AI to answer. The harder, more valuable judgment is deciding when it must not, and building that refusal into the control flow rather than hoping a prompt holds it. Defining the grounding score in the open, setting the threshold for the cost of being wrong, and naming the extractive ceiling instead of papering over it: that is the discipline that lets a regulated organisation put an assistant in front of its people at all.

See it run.

The answers and the abstention above are real output from the assistant over Origin's policy library. The full console, where you can ask anything and watch it cite or abstain, runs as a working application.

See the live governance demo →The full console runs on my machine. Ask me for a 90-second walkthrough.