🔀 Your Agent Is Paying an LLM to Return a Boolean

Count the model calls in your agent loop. Most of them do not write anything.

Is this ticket urgent. Which tool should run next. Is this shell command safe. Is the task finished. Should this go to the cheap model or the expensive one. Every one of those is a boolean or an enum. And the usual way to answer them is a generative model, a prompt that ends in "respond with only YES or NO", and a parser that hopes the model listened.

It works. It is also the wrong tool for the shape of the question.

The glue code nobody talks about

If you have built an agent harness, you have written this code. A prompt that begs for a single word. A regex or a strip().upper() == "YES" check. A retry when the model answers "Yes, because the deploy failed twice..." and your parser chokes. A second retry when it wraps the answer in markdown.

The backend version of this mistake is familiar. It is calling a full remote service for something a lookup table answers. Or running LIKE '%x%' where an equality check on an indexed column would do. The capability is there, so you use it, and you pay for the generality on every single call.

In an agent loop that cost compounds. Each step is another round trip to a model that is built to write paragraphs, asked to write one token.

What Jev does differently

Last week LangChain published a post, Building a harness with Jev, by Sydney Runkle and Hunter Lovell. Jev comes from TypeSafe AI and they call it a "System One model". It does not generate text. You send it a state (the context) and a set of typed questions, and it returns typed answers with a probability attached.

Three question types:

• Choice picks one of several options, with a probability for each.
• Score rates against ordered levels like low, medium, high.
• Noul is a plain yes or no.

Adding more questions to one call barely moves the latency. You pay only for the tokens of the extra questions. So one call can answer "is it urgent", "which team owns it" and "how risky is it" at once.

The basic call from the post:

from langchain_typesafe import Noul, TypeSafeClassifier

classifier = TypeSafeClassifier()

response = classifier.invoke({
    "state": (
        "The deploy failed twice and customers are seeing 500s. "
        "Can someone look now?"
    ),
    "questions": {
        "urgent": Noul(
            instructions="Does this need attention right now?"
        ),
    },
})

urgency = response.nouls["urgent"].noul

No "answer with only YES or NO". No parsing. The contract is the type.

Two places it fits in a harness

The post shows two middlewares, both under langchain_typesafe.experimental.

The first is model routing. The classifier reads the request and picks the cheapest model that can handle it:

from langchain.agents import create_agent
from langchain_typesafe.experimental.middleware import (
    ModelChoice,
    ModelRouterMiddleware,
)

router = ModelRouterMiddleware(
    choices={
        "fast": ModelChoice(
            model="openai:luna",
            criteria="Direct lookups, extraction, and localized changes.",
        ),
        "powerful": ModelChoice(
            model="openai:sol",
            criteria="Architecture and high-stakes decisions.",
        ),
    },
    instructions="Choose the least costly model that can complete the task.",
)

agent = create_agent("openai:gpt-5.6-luna", middleware=[router])

The second is an auto-mode guardrail. Before a tool call runs, the classifier decides whether it looks risky:

from langchain.agents import create_agent
from langchain_typesafe.experimental.middleware import (
    AutoModeMiddleware,
)

guardrail = AutoModeMiddleware(tools=["bash"])

agent = create_agent("openai:gpt-5.6-luna", middleware=[guardrail])

If that sounds familiar, it is the same idea behind the auto-mode classifiers in Claude Code and Cursor. Something small and fast sits in front of the tool call and decides whether a human needs to see it.

The probability is the real feature

A generative yes/no gives you a string. You can ask the model how confident it is, but that is just more generated text, and it is not a number you can put a policy on.

A calibrated classifier gives you a probability. That turns a prompt into a routing policy you own. The post does not show how to read the probability off the response, so here is the shape of the policy with the field access left abstract:

CONFIDENT = 0.80

def route(answer: bool, p: float) -> str:
    # p = probability the classifier assigned to its answer
    if p >= CONFIDENT:
        return "act" if answer else "skip"
    # not sure: pay for the big model, or ask a human
    return "escalate"

That 0.80 is now a knob. Log every decision with its probability, look at the escalation rate and the error rate, move the threshold. You cannot do that with a prompt. You can only reword it and hope.

This is the actual shift. The cheaper model is the small story. The bigger one is a split between two workloads that agent harnesses have been treating as one. Classification wants a typed answer and a confidence. Generation wants text. Different jobs, different tools.

The honest trade-off

I have not run Jev in production. Everything above is about the pattern, and here is what I would want to check before trusting the product.

The numbers are the vendor's best case. TypeSafe AI claims "up to 200x faster inference and 400x lower cost than comparable LLMs on classification tasks". "Up to" on their benchmarks is not your workload. Replay a week of your own agent decisions through it and measure.

Calibration is the whole pitch, and it is a claim. The threshold logic only works if 0.9 actually means right about nine times in ten on your data. If it does not, the knob is decoration. Checking that needs a labeled sample of real decisions. Most teams do not have one, and building it is the real cost of adopting this.

It is experimental. Both middlewares live under experimental. New vendor, new model, sitting in the critical path of every agent step. Plan for the API to move.

The guardrail has the usual blind spot. A classifier that flags risky bash knows what "risky" looks like in general. It does not know your repo. A bulk delete in a generated folder is harmless for you and scary in general. A routine-looking command against the wrong database is the opposite. Same problem every auto-mode classifier has.

It does not replace the LLM. The post says so itself: Jev is not a drop-in replacement for an LLM. Anything open-ended still needs a generative model. You are adding a second model to operate, monitor and version, not removing the first.

Where I would start

Not with the guardrail. Start with the cheapest, most reversible decision in your loop: model routing, or "is this task done". Wire the classifier in shadow mode next to your current LLM check, log both answers with the classifier's probability, and compare them for a week. If the two agree on the easy cases and the probability drops on the cases where they disagree, calibration is doing its job on your data. Then move the threshold into the live path.

If they disagree randomly, you learned that cheaply, and your old regex is still there.

How many of your agent's LLM calls are really classification, and have you tried splitting them out yet?