Put the Language Model Last

Every voice agent stack I've looked at this year hands the language model the whole job. It decides when to speak, what mood to take, whether to look something up, how long to go, and then it also writes the words. That's the slowest and most expensive component in the pipeline being asked to do the fastest and cheapest part of a conversation, which is reading the room. People don't work that way. You take in how someone said something before you open your mouth, and the taking-in is nearly instant. The talking is the slow part.

System 1 decision models have been all the rage lately. Typesafe AI's Jev set the pattern: a model that doesn't generate text at all, just answers closed-set questions with calibrated probabilities, fast enough to sit in front of every decision an agent makes. On Wednesday (Oct. 1st) Cloudflare released Clef, its own family of decision models, open weights, Jev-API compatible, and hosted on Workers AI. Clef-flash runs on a 9B backbone and Cloudflare quotes a median under 40 milliseconds. I wanted to see what it could actually do, and a voice agent is the hardest place to put a model like this, because every millisecond is audible. So I spent the weekend building one around it: a classifier reads each utterance, plain code turns the labels into a plan, and only then does a language model write a reply to fit it. Here's what that bought me and what it didn't.

The Shape of It

Deepgram Flux handles speech-to-text and turn detection. Clef-flash on Workers AI labels each utterance. A rules planner turns labels into a plan. Any model through the Vercel AI Gateway writes the reply, and Cartesia speaks it. The whole thing runs as one Durable Object per person on a Cloudflare Worker, using the Agents SDK's voice kit for the browser audio plumbing.

A Clef call is one request: a state object, a map of questions, and back come answers with probabilities. Three question types. noul is yes/no and returns the probability of yes. choice picks one of named options and returns a probability for each. score is an ordered scale and returns a probability-weighted level.

const response = await ai.run(`@cf/cloudflare/${model}`, {
  model,
  state: { recent_turns: recentTurns, utterance },
  questions,
});
// answers: { needs_lookup: { type: "noul", noul: 0.88 },
//            intent: { type: "choice", choice: "ask_info", probabilities: {...} },
//            openness: { type: "score", score: 2.1 } }

The interesting part is what the classifier is asked. Clef is closed-set, so it only scores the options you give it, and it decides from the wording of your instructions. The questions are the product. I ended up with 45 of them in three sections.

Two of them, so you can see what a question looks like:

needs_lookup: noul(
  "Would a good answer need something checked rather than known: anything current, local, or specific to a place, product, service, tool, or document? Also yes when the speaker asks the assistant to search, check, find, or look again. No for general knowledge, opinions, or talk about the conversation itself.",
),
openness: {
  type: "score",
  instructions: "How open is the speaker, right now, to hearing a view that differs from their own?",
  criteria: ["Closed or defensive", "Guarded", "Open", "Explicitly asking to be challenged"],
},

The planner is where the architecture earns its keep, and it's nothing clever. It's thresholds. A label only drives behavior above a confidence bar; below it, the plan is neutral and the model just talks. The output is a directive the language model sees in its system prompt: what kind of reply, how long, whether to ask a follow-up, whether to push back now or hold it, what to avoid, and how the voice should sound.

const plan = {
  act: "respond",
  length: "short",
  followUp: "none",
  pushback: "hold",
  avoid: ["naming their emotion out loud"],
  notes: ["They sound sure of themselves. Engage with the substance, no reassurance."],
  delivery: { speed: 1, volume: 1, emotion: "content" },
};

The model gets that directive plus a one-line read of what the classifier saw, with the numbers ("feels confident 77%; neutral; medium energy; guarded") and a standing instruction to let it shape the reply without ever bringing it up. The plan does the steering; the read gives the model texture. Keeping those two things separate turned out to matter more than I expected.

The Planner Is Code

The first thing you notice building this way is that the conversation policy becomes something you can test. Thresholds, intent groupings, the fallback to neutral, the rule that says no two questions in a row: all of it is TypeScript with unit tests, and all of it is tunable by editing the wording of a question. When the "confused" question scored 0.97 on "Actually, yes, it does," I didn't retrain anything or rewrite a 2,000-token prompt. I gave the question examples of what confusion sounds like and what it doesn't, and the false positives went away.

Even the order of lines in the directive is a tunable. The follow-up rule goes last, where the model weights it most, because when it sat in the middle of the list the model asked a question anyway on most of the turns it was told not to.

// Last, where the model weights it most.
if (plan.followUp === "ask") {
  lines.push("- End with one specific question about what they said.");
} else if (plan.followUp === "none") {
  lines.push("- End on a statement, not a question. No question anywhere in this reply.");
}

That forces a discipline you don't get when one prompt does everything. Every question costs classifier time, so a question earns its place only if the planner does something different because of the answer. Nobody prunes a mega-prompt with that kind of rigor, because there's no cost signal telling you to. Here there is one. My approach was to ask as many questions as I could think of, see which ones the planner actually used, and prune back. Then I added a whole new set and split them across parallel requests instead of one call. The sweet spot came out around 12 to 15 questions per call. Past that, the call slows down enough to miss the budget; split well, four calls in parallel cost about what the largest one costs.

export const PERCEPTION_SHARDS = {
  doing_and_talking: { ...DOING_QUESTIONS, ...TALKING_QUESTIONS },
  pleasant: PLEASANT_QUESTIONS,
  unpleasant: UNPLEASANT_QUESTIONS,
  scales: FEELING_SCALE_QUESTIONS,
};
 
const results = await Promise.allSettled(
  shards.map(([, questions]) => askClef(ai, state, questions, { timeoutMs })),
);
// A late shard leaves its own answers neutral. Only if every shard fails
// does the whole turn fall back.

The split only works if the buckets are right, though. Questions that get decided together need to travel together, so a late group leaves a coherent hole rather than a random one.

Labels Shape, They Don't Gate

The design rule I'm most glad I made early: nothing acts below a confidence threshold, and the fallback is always "say something normal." The classifier is an overlay on a plain voice chat, not a link in a chain.

A label only drives behavior when the model is confident in it. Otherwise the plan falls back to neutral: acting on a wrong label feels worse than being plain.

That's the comment at the top of the planner, and it's the whole philosophy. Here it is doing its job on the one decision that matters most, whether to speak at all:

const quiet =
  !options.typed &&
  !options.answeringQuestion &&
  (intent === "backchannel" || intent === "thinking_aloud") &&
  labels.expectsReply < CONFIG.planner.quietBelow &&
  labels.isQuestion < 0.5 &&
  bare; // a word or two, carrying no pleasant feeling
if (quiet) return { respond: false, ... };

Every condition has to line up before the agent stays silent. Any one of them unsure, and it talks. I got to test that property harder than I planned. At one point the classifier dropped out entirely for a long stretch (a quota on Workers AI's free tier, which the weekend's experimenting burned through faster than I'd budgeted). The conversation kept going. Replies were a little more generic and a little faster, since there was no perception to wait on, and the persona carried the rest. Most agent pipelines fall over when a component vanishes. This one degraded to the thing it was built on top of. If you're adding a classifier to a voice loop, that's the property to design for first: it should make good moments better, and its absence should cost you nothing but those moments.

The Talker Can Be Small

Once understanding happens up front, the language model's job shrinks to writing one or two sentences that fit a plan. A fast model is enough for that. I ran four through the same prompts.

Sonnet had a little more depth to it, the sense of more experience behind the reply, but for a conversation it wasn't better. Haiku was fine, because the job was a conversation and the plan had already scoped it. If the topic had been technical, maybe the gap would show. It didn't here, and Sonnet was adding most of a second to every turn for it, plus the cost. Not worth it for a difference you have to listen for.

And you don't have to choose once. The classifier has already read the turn before the talker is picked, so the plan can route it: a technical question or a request for real advice goes to Sonnet and eats the extra time where depth earns it, and everything else goes to Haiku. Model selection becomes one more field in the plan, decided per turn from probabilities you already have, instead of a global setting you argue about. The architecture is what makes the cheap model viable, and the cheap model is what makes sub-second voice possible. Those two facts are the same fact.

Where the Classifier Is Actually Good

Not intent. I gave Clef an 18-way choice of what the person was doing and the top option rarely cleared the bar on its own; probability smeared across too many options that are legitimately close. Grouping related intents (asking, sharing, directing, responding) and thresholding the group helped, but the honest answer is that a closed-set model with that many options is going to be unsure a lot, and the planner should treat "unsure" as the normal case.

export const INTENT_GROUPS = {
  asking: ["ask_info", "ask_advice", "task", "repeat", "about_agent"],
  sharing: ["vent", "self_disclosure", "story", "chitchat", "thinking_aloud"],
  directing: ["correction", "complaint", "challenge", "instruction"],
  responding: ["answer", "backchannel"],
  closing: ["closing"],
};
// The top intent counts on its own above 0.4. Below that, the strongest
// group has to clear 0.55 together, and its top member wins. Otherwise: other.

Stance. Open or guarded. Venting or asking for a fix. Expects a reply or just acknowledging. These are binary questions with concrete descriptions, and the probabilities were sharp enough to act on.

Feeling. Confident, curious, confused. Same story: one question each, each with a plain description of what it sounds like, and the answers were usable. They're also exactly the things a language model handles badly when they're dropped into the prompt bare, because when you tell an LLM "the user is frustrated," it tends to say "I understand you're frustrated," which is the worst possible move. What fixed that was pairing the read with a plan ("they're venting, don't offer fixes, one acknowledgment at most") and an explicit rule that the read is for shaping, not saying. The classifier reads the emotion; the model is told what to do with it and told not to narrate it.

When to disagree. This is where the honesty design lives. I wanted something that disagrees when a friend would, and that means knowing when the person is open to it. The openness score and the "questionable claim" question feed a pushback field in the plan: now, hold, or none. Disagreeing at the wrong moment is as bad as never disagreeing, so the planner decides when, not whether.

Latency Is About Overlap, Not Speed

The biggest latency win had nothing to do with making any component faster. Flux emits an eager end-of-turn event when it thinks the person has probably finished, before it's sure. Most stacks ignore it. I use it to start the classifier and, once that looked safe, to start drafting the reply too. If the person keeps talking, Flux sends "turn resumed" and the work is thrown away.

#onEager(connection, transcript, confidence) {
  this.#speculator(connection).start(transcript, (signal) =>
    this.#prepare(transcript, history, { signal, startReply: CONFIG.speculation.reply }),
  );
}
 
#onResumed(connection) {
  this.#speculator(connection).cancel(); // aborts perception and the draft
}
 
// When the turn really ends, the work is claimed if the transcript still
// matches what it was started for. Otherwise it's aborted and done fresh.

That means the classification and often the first sentence of the reply happen during the pause people take anyway. On a good share of turns the reply started before the final end of turn, and the best cases had audio coming back within a few hundred milliseconds of the person stopping. The classifier-fired filler works the same way: when "needs a lookup" is high, the agent says "One second" the moment the person stops, before the language model has run, and the search happens behind it.

Speculation has a bill. Every wrong eager guess burns the classifier calls and a model call. Deepgram's own docs warn eager mode can raise LLM calls by half or more, and I'd believe it; it's where my free-tier budget went. The trade is still worth it for voice, because the thing you're buying is the one thing users feel directly. But set the eager threshold from the log, not from a guess.

Two Tiers of Reaction

The design has a reflex layer and a considered layer. Reflexes fire from the classifier in milliseconds: the filler before a search, silence on a backchannel, a short "Hmm" before a hard question. Considered responses come from the model. The experiment was where the line sits.

The laugh taught me. The persona lets the model write a [laughter] tag inline, which Cartesia renders, and it did that naturally when something was funny. The reflex laugh, the pre-rendered one meant to fire before the model has said anything, needs the classifier to be confident the person was joking, and it never was. Humor is a judgment call, and a closed-set model asked "is this a joke" gives you a shrug. Backchannel silence, on the other hand, fired on exactly the right turns ("Makes sense," "Sounds about right"), because "did they add anything" is a question with a crisp answer.

Reflexes belong on questions the classifier can be sure about. Everything else belongs with the model, and the persona is how you get it there.

Guardrails Without the Round Trip

The first version scored each sentence before it was spoken: does this flatter, does it only agree, is this stock sympathy. The idea was a guardrail against sycophancy. In practice it found nothing across every sentence it checked and cost a classifier round trip per sentence, which you can hear. I pulled it the same day. Honesty moved into the character (a fixed core the person can't configure into agreement) and the measurement moved to after the turn, where it's logged and costs nothing.

You obviously still want to keep the model from doing something stupid. The way to do that is persona work, a proper plan, escalation to a human when the stakes call for it, and evals. Not a second model checking the first one's output on every turn, whether that's an LLM judging an LLM or a classifier scoring each sentence. That's latency for almost no benefit, and in a voice loop you hear every millisecond of it. Evals are where the output gets judged: the log has every label, plan, and reply, so you score the conversation after the fact, find the turns that went wrong, and fix the persona or the plan that let them through.

The classifier still guards the live path; it just does it before the model runs instead of after. You already have its probabilities in hand, so use them: a high "questionable claim" escalates to pushback in the plan, a low openness score holds that disagreement for a better moment, and a "casual swearing" read stops the feeling questions from mistaking profanity for anger and slowing the voice down. That's a guardrail that costs nothing extra, because the call was already being made.

Guard the input, eval the output. A second model checking the first one on the live path is latency with a good excuse.

The Hard Part Is the Audio Loop

The models were the easy part of this. Clef, Flux, Cartesia, and the gateway all did what their docs said. The hard part was owning when sound is allowed to happen. The voice kit holds short sentences until more text arrives, decides when a turn ends on its own, and owns the audio queue. I worked around the first two in a day. The short-sentence hold, for instance, is beaten by handing the kit a structured part after the text, which makes it flush:

export function* forceOut(text: string): Generator<StreamPart> {
  yield { type: "text-delta", text: `${text} ` };
  yield { type: "flush" };
}

The audio queue is the one I couldn't work around, and it's the one that matters most. The kit waits for a complete sentence from the model before it sends anything to the voice, so first audio is gated on the model finishing a thought, not starting one. You can hear it. Cartesia can take text as it streams and start speaking after the first few words, which is exactly what the kit won't let me do. Replacing the server half of the kit with my own loop that streams tokens straight to Cartesia is the only logical next step, because it's the one change that cuts time to first audio and makes the whole thing feel more natural at the same time.

What I'd Tell You to Do

There are real-time voice agent platforms you can buy, and if you want a voice agent next week, buy one. The reason to build something like this is that you don't want a black box. You want to see every label, every plan, every timing on every turn, and you want to be able to change the policy by editing a question instead of filing a ticket. It's also a lot easier to eval. When every turn is logged with what the classifier saw, what the planner decided, and what the model said, you can replay a conversation against a changed threshold or a reworded question and know whether it helped, instead of guessing from vibes. A black box can't give you that. If that's you, here's what I'd do.

Stop asking the language model to run the conversation. Put a decision model in front of it, ask it the questions a plan actually depends on, and make the fallback for every unsure answer "be normal."

Start your work at the eager end of turn and accept the bill. The pause the person takes anyway is the cheapest latency you'll ever buy back.

Use a small talker by default. The plan is doing the thinking, so the model's job is one or two good sentences, fast. Let the plan route a turn to a bigger model when the question earns it.

Guard the input, eval the output. Persona, plan, and human escalation keep the model in line on the live path, driven by probabilities you already have. Evals on the log catch what slipped through. Don't put a second model in the loop to check the first.

Budget your time for the audio loop, not the models. The models work. Owning when sound happens is where the experience is decided.

Clef is what made this worth trying. A model that returns a probability in tens of milliseconds, over a schema you wrote in plain English, is a different kind of building block than an LLM, and I think it's the one voice agents have been missing. The language model is the wrong component to put in charge of a conversation. It's the right one to put last.