Case studies

Ask with a form and fixed choices first, then let a model choose the follow-up questions inside limits set in code

Because the user knows their system and does not know the law, so the questions have to come from a fact schema written from the regulation text.

Measured bythe share of started intakes that reach a memo, and the share of 'not sure' answers per question

Decide which regulations, roles and risk tiers apply with rules in code, not a model

Because the same facts must always give the same result.

Measured bya benchmark of test products with known correct answers

Quote and link the clause in the official text on every row of the memo

Because a person who is not a lawyer has to be able to check each row.

Measured bythe share of rows that users mark as disputed, and hand labels of the written meaning and action

Route risky conversations to a human with a traffic-light escalation protocol

Because an assistant answering in the company's name cannot afford a confident wrong answer.

Measured byrouting correctness: the share of conversations that end in the right place, with the assistant or with a person

Convert every production error into a permanent CI test case

Because prompt and model changes silently reintroduce old failures otherwise.

Measured byrecurrence of known failures: a failure that has a test must not appear again in a later release

Isolate tenants with schema-per-tenant Postgres

Because one business's data must never surface in another business's answers, and each business's data has to be exportable as a unit.

Measured bya business's complete dataset exports as one schema

Hold the model constant and vary only the interface

Because otherwise interface effects are confounded with model quality.

Measured byaction accuracy on a 73-item hand-labeled eval set

Ground every claim with two independent judges (Claude Opus 4.7 and GPT-5)

Because a single judge bakes one model's bias into the results.

Measured bysubstring grounding rates per judge, 99.3% integrated vs 93.4% chat

Keep the eval run that failed, together with the fix and the run after it

Because a scorecard is only useful if it can catch a problem.

Measured byone recorded case: industry classification went from 32.9% to 94.5% after the fix

Classify every query into five categories before retrieval

Because a cancellation request must never get a how-to answer, and nonsense should not waste retrieval compute.

Measured byeach category routing to its own purpose-built response path

Gate answers behind two reranker confidence thresholds

Because generating from weak context erodes trust faster than asking a clarifying question.

Measured byfaithfulness and relevancy scores across 7 experiments (4.80/5 and 4.76/5)

Run a self-critique step after generation

Because the cheapest place to catch a partial answer is before the user sees it.

Measured bythe critique verdict deciding whether human escalation is offered

Score five dimensions independently and let the weakest set the level

Because a feature is constrained by its weakest dimension, not its average.

Measured bythe min-rule stated explicitly in the published rubric

Give every level both a user-visible signal and a team signal

Because a feature can look adopted while being operationally fragile, and the reverse.

Measured byboth signal columns present for all five levels

Make output quality one dimension of five, not the whole score

Because accuracy can improve while adoption stays flat.

Measured byfour of the five dimensions cover something other than output quality