The most useful thing this system does is refuse to answer.

On enterprise schemas, frontier models collapse quietly.

The original Spider text-to-SQL benchmark is largely solved: top models score around 86%. Spider 2.0 is a different world. Built on real enterprise warehouses with hundreds of columns, ambiguous metrics and certified join paths, it drops frontier models to roughly 10–17% (GPT-4o ≈ 10.1%, o1-preview ≈ 17.1%).¹ What matters is the shape of that collapse. The model does not error out. It returns SQL that runs and produces a number that is simply the wrong number: a returns table joined on the wrong key inflating revenue, a missing fiscal-period filter double-counting the quarter, a non-sanctioned definition of "revenue." In a MAS-regulated bank, a wrong-but-plausible figure in a regulatory return is a material-misstatement risk, not a UX bug.

A model that writes SQL is also a model that defines revenue.

The seductive architecture is: English in, model writes SQL, SQL runs, number out. It fails on contact with a real schema in a handful of specific, knowable ways, and each one returns a number that looks right.

• Uncertified joinInflates the headline. fact_sales and fact_returns share product_id, so joining on it fans rows out many-to-many and double-counts revenue. The only certified path is ON order_id, and the model has no way to know that.
• Missing filterDouble-counts the period or the entity. Drop the mandatory fiscal_period filter and "this quarter" silently sums four. Drop legal_entity and three legal books consolidate into one number no one asked for.
• Wrong grainCounts the wrong thing. COUNT(order_id) over a line-grain fact counts order lines, not orders. The contract requires COUNT(DISTINCT order_id) at order grain. That one-word difference inflates the count ~2.6×.
• Off-catalog metricInvents a definition. Asked for "profit margin" with no certified cost metric, a baseline quietly substitutes list price for cost and ships a confident, meaningless percentage.

None of these throw an error. The naive system is at its most dangerous precisely when it is most confident.

The model proposes and deterministic code decides.

The whole design turns on one line: the model never decides whether a query runs. It is allowed to do exactly two things. It translates English into a candidate SQL string, and later phrases the human-readable explanation. Between those two acts sits a deterministic referee that parses the candidate and checks it, clause by clause, against a published, versioned semantic contract. An off-catalog metric, an uncertified join, a wrong grain or a missing mandatory filter makes the system abstain and name the exact clause it failed. A sensitive column routes to a human data steward. Nothing is invented. Absent evidence, the answer is "abstained."

Here is the actual decision. This function owns the verdict. It is deterministic, it reads only the contract, and the model's output is an input to it, never an authority over it.

// The referee: parse the model's candidate SQL, then check it clause-by-clause
// against the versioned contract. Returns ALLOW / ABSTAIN / ESCALATE.
function referee(q) {
  const p = parseSQL(q.candidate_sql);          // tables, joins, filters, grain

  // A. metric must be on the certified catalog (else ABSTAIN)
  const met = detectMetric(q, p);
  if (met.id === null)
    return verdict("ABSTAIN", q, p, checks,
      `metrics: '${met.reason}' is not in the certified catalog`);

  // B. every JOIN must match a certified {tables, key} pair (else ABSTAIN)
  for (const j of p.joins) {
    const k = resolveJoinKey(j.on, p.aliases);
    const ok = CONTRACT.certified_joins.some(cj =>
      pairMatches(cj, k) && cj.on === k.lc && cj.on === k.rc);
    if (!ok)
      return verdict("ABSTAIN", q, p, checks,
        `certified_joins: ${k.lt}↔${k.rt} must join ON ${certKey}`);
  }

  // C. sensitive column / table -> human data steward (ESCALATE / HITL)
  if (touchesSensitive(p))
    return verdict("ESCALATE", q, p, checks,
      `sensitive_columns: routes to a data steward (HITL)`);

  // D. required grain (order_count needs COUNT(DISTINCT order_id))
  if (met.id === "order_count" && !isDistinctOrder(p.select))
    return verdict("ABSTAIN", q, p, checks,
      `required_grain: needs COUNT(DISTINCT order_id) at order grain`);

  // E. mandatory filters on any fact table touched (fiscal_period, legal_entity)
  if (touchesFact(p))
    for (const f of CONTRACT.mandatory_filters)
      if (!present(f, p.where))
        return verdict("ABSTAIN", q, p, checks,
          `mandatory_filters: query is missing '${f}'`);

  // all clauses satisfied -> the contract certifies this query
  return verdict("ALLOW", q, p, checks, null, met.id);
}

The drafting step is fenced just as hard. Every clause the model writes must point at the contract evidence it came from, and where there is no evidence it writes the literal words "abstained, no evidence." The cite-or-abstain rule is enforced in code, not asked of the model.

One question, two answers, and only one is allowed to ship.

Take the flagship pair: "What was net revenue for Q3 2025 in the US entity?" A naive baseline joins sales to returns on product_id, the wrong key, so the returns rows fan out and double-count. It ships a confident $9,904,220. The referee parses that same candidate, finds the join does not match the only certified fact_sales↔fact_returns path (which is ON order_id), and abstains, naming the clause. The certified answer, reachable only through the contract, is $4,128,500. The gate would rather return nothing than return the inflated figure.

What the fixed bank shows, and what it does not.

Where these come from, flatly. The warehouse, the contract and the 25-question bank are synthetic. A seeded generator lets any third party re-run it and reproduce identical data, identical verdicts and identical ledger hashes. So these figures prove the mechanism: that a deterministic contract check, run on candidate SQL, catches the documented Spider-2.0 failure shapes and prevents the wrong number. They are not a claim about accuracy on a specific real warehouse.

The honest limits, stated plainly. A production-grade static SQL analyzer over an arbitrary certified join graph is far harder than this demo's bounded grammar. Full dialect coverage, CTEs, subqueries and window functions are real engineering this does not attempt. The result numbers for allowed queries are precomputed, not executed live, because the build deliberately ships no SQL engine. And the 25-question bank, although seeded from real documented failure modes, is a fixed set. It demonstrates the trust-boundary concept on parseable candidate SQL, not detection coverage of every way an LLM can be wrong. I would rather say that out loud than oversell it.

Every verdict is logged, and the log can be broken on purpose.

Each ALLOW, ABSTAIN and ESCALATE, along with each steward approval, is written to an append-only, hash-chained ledger. An entry's hash covers its contents and the previous hash, so altering any record after the fact breaks the chain at exactly that row. The live demo includes a Tamper button that flips a logged decision without rehashing. Verify then reports the break at the precise sequence number, and Replay re-derives all 25 verdicts from the contract and finds the one that no longer reproduces.

seq        : 2
decision   : ABSTAIN        contract : v3.2.0
subject    : Q01  (net revenue, Q3 2025, US)
clause     : certified_joins: fact_sales↔fact_returns must join ON order_id
prev_hash  : b4df01d2
hash       : 02a6293e   ← = fnv1a(prev_hash + seq|qid|decision|clause|contract)

See it run.

The live application leads with the two adversarial pairs, putting the baseline's wrong number beside the gate's abstain, then lets you run any of the 25 questions and the full bank, with the hash-chained ledger, Verify, Replay and Tamper on the right.

1. Lei et al., Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows, xlang-ai / The University of Hong Kong, ICLR 2025 (frontier-model collapse to ~10–17% on enterprise schemas vs ~86% on Spider 1.0).
2. Snowflake Cortex Analyst / Open Semantic Interchange and dbt MetricFlow semantic-layer documentation, on the pattern of a governed metric/semantic layer between natural language and the warehouse.
3. Databricks Genie / Metric Views documentation, on certified metrics and governed self-service analytics over a lakehouse.

All data, the semantic contract and the question bank in the linked demo are synthetic and clearly labelled as such; figures illustrate the control mechanism, not a real-world result.