Making an AI sales assistant answer sooner with a decision model.

The context

The sales assistant answers a manufacturer’s technical questions from its product documents. Before it answers, a router picks one of three answer models: light for greetings and simple lookups, standard for most questions, deep for the hardest. Easy questions on the cheaper model save money, but the router runs on every question, before anything else can happen.

The problem

Over the pilot’s first 27 answers, with Claude Haiku as the router, choosing a model took 2.1 seconds at the median and 4.3 seconds at the 95th percentile. The first word of an answer arrived after 5.3 seconds. Speed was the weakest part of the experience, and the router was a large share of the wait.

The router is under 1% of what a question costs, so the problem was time, not money. Haiku is a language model doing a classifier’s job: it reads a prompt, writes its choice as JSON for code to check, and in testing it timed out on 2 of 141 calls.

The solution

I added TypeSafe’s Jev as a second router that can replace Haiku with one setting. Five choices shape it:

• Jev is a decision model built to pick from fixed options, so it returns a choice and a confidence in about 160 ms, with no text to parse.
• Jev proposes and code decides, so an unsure or unsafe proposal falls back to the standard model.
• There’s no second router behind it, so if Jev fails, standard answers and an outage costs at most 2.5 seconds.
• A super admin switches the router per environment and every change is kept, so a preview can run Jev while production stays on Haiku.
• Every route logs the router, its latency, cost and confidence, so a settings page compares the two over 30 days.

I designed the switch, the code rules and the evals, and ran the comparison and the preview. Coding agents wrote the code from plans I approved, and I reviewed every change before it merged.

How it works

A question goes through five steps:

1. 01
   Signals. Code reads the question first: its language, product names, comparison words and price words.
2. 02
   Two questions. One request asks Jev which tier the question needs and what kind of question it is. Jev sees what Haiku saw: the question, the previous question, the first 400 characters of the previous answer and the signals. The options reuse the Haiku prompt’s descriptions, so the two can be compared fairly, and standard comes first because Jev leans towards the first option.
3. 03
   The rule. Code keeps Jev’s proposal only if it passes the rules below. Otherwise the question goes to standard.
4. 04
   The answer. The chosen model answers from the documents, as it would with Haiku as the router.
5. 05
   The record. The audit log keeps the router, its latency and cost, Jev’s confidence and any overruled proposal. Jev runs outside the AI gateway, so code prices each call from TypeSafe’s list price.

When code overrules Jev

Code keeps Jev’s proposal only when it agrees:

• Standard is accepted as it is, as the safe default.
• Under 0.5 confidence in the tier, the question goes to standard.
• Light is allowed only for small talk, out-of-scope questions, lookups and follow-ups.
• An error, an answer outside the options or no reply within 2.5 seconds also means standard.

When code overrules Jev, the audit log keeps the tier it proposed and its confidence next to the tier that answered. Try the rule:

During an outage, light and deep questions both get standard: easy ones cost more, and the hardest get a less capable model. Waiting for Haiku as well would have added its own seconds to the wait that was already the problem.

Evals

The eval grades each route by exact match against a labelled tier. It refuses to start without a TypeSafe key, so a silent fallback to Haiku can’t pass for Jev. I ran the same 47 labelled questions, in three languages, through each router three times: 141 calls each. Haiku’s run was two days earlier, with the same set and settings.

All six misses are a short follow-up asking about another product after a comparison. It needed deep and got standard, the safe direction. In five, Jev’s confidence was under 0.5 and code sent the question to standard; once, Jev chose standard itself. Here the right tier depends on the previous question, not the words asked, and the set has only this one case, so it can’t separate the two routers on accuracy.

The preview run

Then the same 47 questions went through the app’s real chat route on a preview, with Jev switched on. Jev routed all 47, 45 as expected, with no fallback and no deep question sent to light. Its median was 158 ms on Vercel.

The two misses were that follow-up again, and the log showed what the offline eval couldn’t. Jev had proposed light, at confidence 0.20 and 0.35, for a question it labelled a comparison. Both code rules would have stopped it. Standard answered instead, and the check found all 12 of its claims supported.

The router barely moves what a question costs. Priced over a million questions at the preview’s mix, Jev comes out about 5% cheaper, and almost all of that is its two misroutes sending deep questions to the cheaper standard model: a quality cost, not a saving. The case for Jev is speed.

What production needs

• The client’s approval. With Jev on, each question, the previous question and the start of the previous answer go to TypeSafe, a new sub-processor. Production stays on Haiku until the client agrees.
• A measured time to the first word. The 1.2 seconds is routing time saved, not an end-to-end measurement.
• More follow-up cases, tested on new questions rather than this set. The likely fix is a code rule that lets a short follow-up inherit the previous turn’s tier.
• A second preview run, since the first ran once, and then production traffic, of which there is none on Jev yet.

Lessons

Evaluate the whole route, not just the model. Offline, Jev’s misses looked safe. Through the real route, it had proposed light for a hard question, and the code rules caught it. The rules carry the safety, so they have their own tests.

The cheaper call isn’t the saving. The router is under 1% of a question’s cost either way, so the case rests on the time to the first word. And a new vendor is a data decision: the switch, the logging and the comparison are ready for the day the client says yes.

Figures from October 2026: a 141-call eval per router, one 47-question preview run and the pilot’s first 27 production answers. The client, its products and its conversations are withheld.