EN English ES Español
AI Strategy

Where Decision Models Fit in an AI Agent Stack

A fast decision model belongs in front of expensive LLM calls — routing, tool selection, loop-break and escalation. The two-speed agent pattern, with a reference architecture builders can copy.

By Dr. Sanju Abraham 18 min read
A decision model sits as a turnstile in front of the LLM — routing fast cases to direct actions and reserving the slow lane for genuine reasoning.

By Dr. Sanju Abraham, Fractional CTO — 180+ projects shipped, 34 funded companies. Published October 15, 2026.

Key takeaways

  • A decision model belongs in front of the LLM in every agent stack. It handles routing, classification, tool selection, and next-action decisions in tens of milliseconds instead of seconds.
  • The pattern has a shape worth naming: the two-speed agent. A fast decision layer resolves 70–90% of steps on its own; a slow LLM layer only runs when prose or multi-step reasoning is genuinely needed.
  • Five specific placements earn their keep inside an agent today. Tool-selection, intent-routing, loop-break conditions, confidence-gated escalation, and final-answer formatting choice.
  • Every major agent framework already supports this shape. LangGraph conditional edges, OpenAI’s tool_choice, Anthropic’s strict tool use, and Mastra’s workflow steps all let a cheap classifier gate the expensive call.
  • The savings compound. Latency drops from seconds to tens of milliseconds on the fast path, cost per step drops by one to two orders of magnitude, and the agent’s behavior becomes testable because the gating logic is typed.

The short answer: In an AI agent stack, a fast decision model should sit in front of expensive LLM calls — classifying intent, selecting tools, deciding whether to loop again, and choosing when to escalate — so the frontier LLM is reserved for the subset of steps that actually need prose or multi-step reasoning. The architectural shape I will walk through in this piece is the two-speed agent: a fast lane (decision model → direct action) and a slow lane (LLM → reasoning or writing), with a typed boundary between them that your tests, logs, and dashboards can see.

Picture a well-run restaurant kitchen. There is a line cook who, for ninety percent of the orders, just reads the ticket and plates the dish — no thinking required. There is also a chef who steps in for the orders that need actual judgment: the allergies, the dietary swaps, the off-menu requests. If every ticket went through the chef, service would collapse by 7 p.m. If every ticket went through the line cook, the picky guests would leave unhappy. The kitchen works because the ticket is read fast and routed to the right station. An AI agent stack is the same shape. The line cook is a decision model. The chef is your frontier LLM. The ticket router is where the architecture has to be designed on purpose. This piece is about how to build that router.

Where does a decision model fit in an AI agent stack?

A decision model fits anywhere an agent has to pick between a small set of next steps. Specifically: when to call which tool, which branch of a workflow to take, whether to keep looping or terminate, when to escalate to a human, and what format the final answer should take. All of those decisions arrive in a shape — Choose, Rate, or Verify — that a decision model handles natively in a single typed call.

The alternative, which most teams default to, is to let the frontier LLM decide all of these in the same call where it also does the reasoning and writes the prose. That works. It also costs 10–50x more per step, adds one to five seconds of latency to every branch, and makes the agent’s behavior non-deterministic in a way nobody wants to debug on a Thursday night.

A decision model in front of the LLM is not a replacement for the LLM. It is a traffic controller. For the general chat-vs-decision framing this piece builds on, see chatbot vs decision model: what’s the difference?

What is the two-speed agent?

The two-speed agent is an architectural pattern where every step of the agent runs through a fast decision layer first, and only reaches the slow LLM layer when the decision model says the step genuinely needs reasoning. The two speeds are:

  • Fast path (10–100 ms, cents per million calls). A decision model classifies the input, picks the tool, scores the risk, or verifies a flag. The agent acts on the typed answer directly — no prose, no reasoning, no second call.
  • Slow path (1–5 seconds, dollars per million calls). A frontier LLM does what only an LLM can do: write a reply, plan a multi-step action, summarize a long document, or reason about an ambiguous situation.

The fast lane handles the bulk of the volume in most real agents. In the support-triage agents I have built with clients, the fast lane carries 70 to 90 percent of steps; the slow lane only fires when the user has asked an open-ended question or the decision model’s confidence falls into the middle band of the three-band rule.

The point is not to eliminate the LLM. The point is to stop paying LLM prices for decisions a cheaper, faster, typed model can make equally well — and to reserve the LLM budget for the places where its reasoning is actually the deliverable. The underlying “System One / System Two” framing for why this split works is covered in what is a System One model?.

THE TWO-SPEED AGENTA fast decision layer gates the slow LLM layer.EVENTuser turn,webhook, cronDECISION MODELChoose / Rate / Verifytool? route? loop?confidence 0.0–1.0high confneeds proseloop break / terminateFAST PATH · 10–100 msDirect tool calllookup, write, update, notify~70–90% of stepsSLOW PATH · 1–5 sLLM reasoning or prosedraft reply, plan, summarizetool chaining with reflection~10–30% of stepsSame agent, two speeds. The decision model decides which lane.
The reference shape of a two-speed agent: a decision model classifies, scores, or verifies each step and sends it down a fast or slow lane accordingly.

Five places a decision model earns its keep inside an agent

Here are the five placements I come back to on almost every agent engagement. Each one is a Choose, Rate, or Verify question against the current agent state — nothing more exotic — but each one, done with a decision model instead of a frontier LLM, cuts a specific class of latency and cost out of the stack.

  1. Tool selection. Before you let the LLM pick which of twenty tools to call, let a decision model do it. A Choice over the tool list, with the user turn and a one-line description of each tool in context, returns the right tool 90%+ of the time in tens of milliseconds — then you only call the LLM to fill the arguments. On OpenAI’s function calling, this is the same shape as pre-pinning tool_choice to the function the classifier picked, instead of leaving it on "auto".
  2. Intent routing. The first step inside almost every agent is: what kind of request is this? Billing question, feature request, outage report, abuse, chit-chat. A Choice across your intent taxonomy sends each request down a dedicated sub-graph with its own tools and system prompt — faster, more predictable, and dramatically easier to monitor than a monolithic agent reasoning about everything.
  3. Loop-break conditions. Agent loops that call tools until “done” are a known tail-latency hazard. A Verify question — “Does the current state satisfy the user’s goal?” — run every N steps as a cheap decision call, catches finished tasks early and prevents the LLM from inventing more work. This is the shape most frameworks call a “conditional edge.”
  4. Confidence-gated escalation. Not every decision deserves the frontier LLM. Run the cheap decision model first; if its confidence is above 0.9, act on it; if it’s in the middle band, escalate the same question to the LLM with the decision model’s answer and reasoning attached as a hint. You pay slow-lane prices only for the subset of steps where the fast lane was unsure — the pattern described in why “70% sure” beats “yes”.
  5. Final-answer formatting choice. Even at the end of an agent run, there’s a decision: does the user want a one-line answer, a bullet list, a code block, a chart, or a form? A Choice across your output formats, driven by the user’s turn and the data you have in hand, picks the right template before you ever ask the LLM to write prose into it — and when the format is “lookup” or “none,” the fast lane answers without the LLM at all.

Those five do not exhaust the surface. They do cover 80% of where I see decision models pay for themselves on real agent stacks.

How much latency and cost do you actually save with a decision-first stack?

You save roughly one order of magnitude on each. The numbers depend on the model, the workload, and the vendor, but the direction is consistent across every real deployment I have measured.

  • Latency. A frontier LLM call on a short input is 1 to 5 seconds. A decision model call — Jev, OpenAI’s structured outputs with a small model, Anthropic’s strict tool use with Haiku — is 70 to 500 milliseconds per TypeSafe’s own numbers. Replacing one LLM call with one decision call on the fast lane removes seconds from the critical path.
  • Cost per step. Decision model pricing is one to two orders of magnitude below frontier LLM pricing per token, output tokens are shorter by construction, and you only pay the LLM on the slow lane. High-volume agents — email triage, ticket routing, moderation — see monthly bills drop by 70–90% when they adopt the two-speed shape.
  • Testability. The fast lane returns a typed value you can assert on in CI. You can golden-set your routing, your intent classification, and your loop-break logic. Prose answers from the slow lane still need judgment-based evaluation — but the agent’s structural behavior is now deterministic in a way plain LLM-agents never are.

Treat the numbers above as directional. Benchmark your own workload before you commit to a redesign; the point of the pattern is that measuring is easy once the typed boundary is in place. The companion piece on reading decision-model benchmarks walks through how to do that without being fooled by vendor-reported figures.

Weighted-average latency and cost drop by roughly 5x when the decision layer handles the fast cases itself. Numbers are illustrative; benchmark your own workload.

What do agent frameworks give you today for decision-model gating?

Every major agent framework already supports the two-speed shape. You do not need a new library; you need to use the one you have on purpose.

  • LangGraph. The whole mental model is nodes connected by edges, and conditional edges let a routing function read state and pick the next node at runtime. That is exactly where a decision model belongs: the routing function calls a Choice or Verify, returns the label, and add_conditional_edges sends the graph down the matching lane. See the official LangGraph docs for the current conditional-edge API; third-party tutorials cover the idea well, but check the version-specific signature before you commit.
  • OpenAI Assistants and Responses API. The tool_choice parameter accepts "auto", "required", "none", or a specific tool name. A decision model picks the tool; your Responses call pins tool_choice to that tool and goes straight to arguments. OpenAI’s function calling guide is the authoritative reference for current behavior.
  • Anthropic Computer Use and strict tool use. The computer-use loop itself is a decision-and-act cycle, and Anthropic’s computer use docs describe strict tool-use primitives you can call with a smaller, faster Claude model as a decision gate in front of the frontier Claude that does the reasoning. Smaller model picks the next action; larger model only runs when the picked action’s own output is a dead end.
  • Mastra. Mastra’s workflow steps and agent networks are a natural fit for the two-speed split — a classifier step at the top of the workflow routes to one of several agent paths. The current API lives at Mastra’s docs; as with every framework in 2026, assume the surface will keep moving and treat vendor-reported behavior as a starting point, not a contract.
  • CrewAI and OpenAI Agents SDK. Both have equivalents — “task routing” in CrewAI, “handoffs” in the Agents SDK. The names change; the pattern does not. A decision model, called before the LLM, decides which crew member or which agent gets the turn.

Hedge on specifics. Framework APIs in 2026 are still moving; the pattern is not. If a framework lets you insert a function in front of the LLM call, you have everything you need to build a two-speed agent inside it.

A reference architecture for the two-speed agent

Here is the minimum shape I ship on a two-speed agent build. Everything else is instrumentation.

  • An ingress node. Normalizes the incoming event into a typed state object. User turn, webhook, cron — all shapes converge on one schema before anything else runs.
  • A decision node. One call to a decision model with a bundle of Choose / Rate / Verify questions against the state. Common bundle: Choice("intent", [...]), Choice("next_tool", [...]), Verify("goal_satisfied?"), Score("urgency", 1, 5). Returns a typed struct with confidence scores on every field.
  • A router. Plain code that reads the decision struct and dispatches: high-confidence routes go straight to the fast lane; middle-band confidences escalate to the slow lane with the decision struct attached as context; goal_satisfied branches to the terminate node.
  • A fast lane. Deterministic tool calls — database writes, API calls, template renders — that do the actual work when the decision model is confident. No LLM involvement.
  • A slow lane. The frontier LLM, with the decision struct and the conversation state as context. This is where you spend your LLM budget on purpose.
  • A terminate node. Formats and returns the final response. The format itself is a Choice (one-line, bulleted, structured, chart, silent).
  • A trace table. Every state transition, decision struct, and lane taken gets logged with the step id. This is the artifact your tests assert on and your dashboards graph. Without it, the two-speed shape is invisible in production.

Build those seven pieces and you have a two-speed agent. The specific framework barely matters — LangGraph, Mastra, OpenAI Agents SDK, a hand-rolled state machine; the shape maps onto all of them. For the general discipline of building an AI feature with a typed boundary at every seam, see how I use a decision model in my own product and the broader AI strategy for SMB owners that survives contact with reality.

Seven blocks, two lanes, one trace. The minimum two-speed shape — ingress, decision, router, fast lane, slow lane, terminate, trace — maps onto any framework.

What NOT to put behind the decision gate

The decision model is a traffic controller, not a thinker. A few things belong in the slow lane every time, and quietly cost you when you try to force them into the fast one.

  • Prose a human will read. Reply drafting, summaries, explanations, anything with voice. Decision models cannot write prose a human will pay attention to.
  • Multi-step planning across unknown subtasks. When the step after this one depends on reasoning about what might go wrong next, that is the LLM’s job. Decision models decide; they do not strategize.
  • Novel category invention. Choice picks from the list you supply. If the right answer isn’t in the list, you need an LLM to help you invent the list — a one-time slow-lane job that then expands the fast lane’s vocabulary for every future call.
  • Trust decisions in production without human review above the gate. Confidence scores are approximate. Calibration isn’t automatic; the three-band rule’s REVIEW band is where humans belong, and the fast lane should hand off there rather than hide the uncertainty.

If a step on your agent fits one of those four, do not try to compress it into a decision call. Let the slow lane handle it. The savings elsewhere pay for the places you leave alone.

Frequently asked questions

Do I need an agent framework to use decision models?

No. The two-speed shape is a pattern, not a library. A plain state machine, a Node.js function, or a Python script with an if/else on the decision model’s output is a perfectly valid implementation. Frameworks like LangGraph, Mastra, and the OpenAI Agents SDK make certain features — tracing, retries, resumability — easier, but they are not required to start. Build the pattern first; adopt a framework when the complexity earns it.

What agent frameworks already support decision-model gating?

All the mainstream ones in some form. LangGraph has conditional edges; OpenAI’s Responses API has tool_choice; Anthropic’s strict tool use is exactly the shape a decision model needs; Mastra, CrewAI, and the OpenAI Agents SDK all expose equivalent routing primitives. Treat vendor-reported behavior as a starting point and benchmark your own workload before committing to a framework-specific API.

When does a decision model REPLACE the LLM in an agent?

Whenever the step’s output is a label, number, probability, or tool choice, and the user never sees the output as prose. Routing, scoring, flagging, selecting — all pure decision model territory. The LLM comes back the moment the deliverable is language a human will read or multi-step reasoning across unknown subtasks. See chatbot vs decision model for the longer version.

What’s a tool-selection decision model?

It’s a decision model whose only job is picking which tool to call next from a bounded list. Instead of asking the frontier LLM “which of these twenty tools fits this user turn?”, you ask a decision model the same question as a Choice across the tool names. The LLM then only runs to fill in the arguments for the pinned tool, which is dramatically cheaper and more deterministic than letting it search the whole tool space.

How much latency do you save in a typical two-speed agent?

Enough to notice. On the fast lane, you replace a 1–5 second LLM call with a 70–500 ms decision call; weighted across typical traffic, average step latency drops by roughly 5x. Treat the exact numbers as directional — your mileage will depend on model, prompt, network, and workload — but the direction is consistent across every two-speed agent I have measured.

Can I use the same vendor for the fast and slow lanes?

Yes, and most teams should. OpenAI’s small models (or structured outputs on the frontier model) handle the fast lane while GPT-5 handles the slow lane; Anthropic’s Haiku-class model fronts a Sonnet-class reasoning step; TypeSafe AI’s Jev gates calls to whichever frontier LLM your product already uses. The two-speed pattern is vendor-agnostic — the shape is what matters.

How is this different from “just use a smaller model for routing”?

The two-speed pattern is about the typed boundary between the fast and slow lanes, not just the model size. A smaller LLM that still returns prose doesn’t give you the testability, the confidence scores, the typed state transitions, or the deterministic behavior that make the pattern pay off. A decision model is the specific shape — Choose, Rate, Verify, with confidence attached — that turns “use a smaller model” into a production-grade pattern.

If you are architecting an AI agent right now and the fast-lane / slow-lane decomposition is starting to feel bigger than one afternoon, that boundary-by-boundary design call is exactly what a Fractional CTO is paid to make. One hour on the right partition can save a quarter of engineering time, and the mistakes in the first six months of an agent stack are the expensive ones. More pieces in this thread live in the AI Strategy archive, including the companion deep-dive on choosing a decision API in 2026.

  • #ai-agents
  • #decision-api
  • #architecture
  • #langgraph
  • #builders
  • #ai-strategy

Need a CTO without the headcount?

Schedule a 30-minute intro call. We'll listen first, recommend second.

Schedule a Fractional CTO call Contact us

Related posts

AI Strategy

Choosing a Decision API in 2026: My Practical Framework

Thirteen decision APIs, five questions, five short picks. The five-question framework a Fractional CTO uses to pick a decision API vendor in late 2026 — plus why Jev stays the default while I A/B test the alternatives.

 AI Strategy

Open-Weight Decision Models: Can You Run Your Own Jev?

Kev, Laya, Tev1 and Cloudflare's Clef put open-weight decision models in reach. Here's the honest tradeoff — privacy and control vs GPU cost, ops work, and license fine print — and the volume math that decides whether self-hosting pays.

 AI Strategy

Calibration Is the Feature Nobody Is Talking About

The decision model industry has started a benchmark war on accuracy. Meanwhile the number that actually determines whether you can automate — calibration — barely gets a slide. Here is why calibration, not raw accuracy, is the under-discussed quality bar in 2026.