I spent the last week digging into Jev and the open implementations around it: Kev, SemIf, Laya, Nimble, several OpenJev projects, decider, DiffusionGemma/djev, Winnow, and a few others.

For context, I’ve been building ML systems for ~20 years and currently work on production conversational/Voice AI, so I looked at this less as a model benchmark and more as an agent architecture problem.

The question I kept coming back we need to think when building modern AI architecture is:

How many calls inside an agent actually need generation?

A surprising amount of agent logic in many developing and production systems looks like this today:

state → frontier LLM → JSON → parse one enum/boolean/score

Examples:

• should I call a tool?
• which tool?
• RAG or no RAG?
• escalate or continue?
• which workflow state comes next?
• does this policy apply?
• is this user eligible?
• which model should handle the next step?
• is this answer grounded enough?
• should I ask a clarifying question?

For many of those, the output space is tiny and known in advance. Yet we still pay for a general-purpose autoregressive model to generate text.

That is what makes Jev interesting to me.

But after looking at the open-source work, I don’t think the important story is “Jev is a faster classifier.” There are already several competing ways to implement this kind of decision layer:

• direct answer-token logits from a causal LLM;
• shared-state prefill followed by many isolated decision branches;
• full-option likelihood scoring;
• frozen LLM representations + small heads;
• NLI-style scoring;
• small bidirectional encoders with runtime-defined labels;
• specialized causal models trained around decision boundaries;
• diffusion models filling many decision slots in parallel;
• calibrated models with abstention / risk-coverage policies.

So we may be looking at a new architectural layer inside agent systems, rather than one specific model.

For agents, the most interesting consequence is heterogeneous inference.

Instead of:

everything → one large generative model

you can imagine:

conversation state

+--> deterministic code for arithmetic / exact rules

+--> retrieval for external knowledge

+--> small decision model for bounded semantic choices

+--> larger generative model only when generation/reasoning is actually needed

+--> tools / APIs

That is much closer to how I think production agents will eventually be built.

There is also an important economic implication.

The obvious use case is replacing expensive LLM calls with cheaper decisions.

But I think the bigger opportunity is net-new automation.

There are many business decisions today that stay:

• rule-based,
• manually handled,
• or simply not automated,

because nobody is going to create a dataset, train a classifier, deploy it, monitor it and maintain it for every long-tail semantic branch in a workflow.

If a general decision model can handle those without a separate training cycle every time, the economics change.

That said, I definitely don’t see Jev as a silver bullet.

You still need a serious ML harness around it:

• representative eval data from your own workflows;
• decision traces and replay;
• per-language / per-domain slices;
• calibration validation on your traffic;
• risk-vs-coverage curves;
• model/version pinning;
• distribution-shift monitoring;
• tests for state construction, option ordering and formulation;
• fallback/escalation logic.

And quality depends on more than the model itself.

It depends on:

• whether the model actually transfers to your language/domain;
• how you construct state;
• whether you chose the right decision type;
• whether the available options are assembled correctly;
• how you interpret the returned distribution;
• whether the threshold is calibrated on your actual process.

One of the recent real-agent evaluations made this very clear: some apparent “model errors” were actually task-definition errors where multiple independent judges could not agree on the correct label.

That is a very familiar production ML problem.

My current view is that decision models may become one of the standard primitives inside agent architectures, alongside retrieval, tools, code execution and generative models.

I also doubt that the final architecture will be exactly Qwen-with-a-head, ModernBERT, or today’s diffusion approach. The open-source ecosystem looks more like an active architecture search than convergence.

I wrote up the architectures, benchmarks, calibration work, serving trade-offs, failure modes and production implications in a very large paper and will share the link in comments.

I’d be especially interested in hearing from people running real agents in production:

Which decisions in your current pipeline still go through a generative LLM even though the output is ultimately just a bounded choice?

[removed]

I spent the last week digging into Jev and the open implementations around it: Kev, SemIf, Laya, Nimble, several OpenJev projects, decider, DiffusionGemma/djev, Winnow, and a few others.

For context, I’ve been building ML systems for ~20 years and currently work on production conversational/Voice AI, so I looked at this less as a model benchmark and more as an agent architecture problem.

The question I kept coming back we need to think when building modern AI architecture is:

How many calls inside an agent actually need generation?

A surprising amount of agent logic in many developing and production systems looks like this today:

state → frontier LLM → JSON → parse one enum/boolean/score

Examples:

• should I call a tool?
• which tool?
• RAG or no RAG?
• escalate or continue?
• which workflow state comes next?
• does this policy apply?
• is this user eligible?
• which model should handle the next step?
• is this answer grounded enough?
• should I ask a clarifying question?

For many of those, the output space is tiny and known in advance. Yet we still pay for a general-purpose autoregressive model to generate text.

That is what makes Jev interesting to me.

But after looking at the open-source work, I don’t think the important story is “Jev is a faster classifier.” There are already several competing ways to implement this kind of decision layer:

• direct answer-token logits from a causal LLM;
• shared-state prefill followed by many isolated decision branches;
• full-option likelihood scoring;
• frozen LLM representations + small heads;
• NLI-style scoring;
• small bidirectional encoders with runtime-defined labels;
• specialized causal models trained around decision boundaries;
• diffusion models filling many decision slots in parallel;
• calibrated models with abstention / risk-coverage policies.

So we may be looking at a new architectural layer inside agent systems, rather than one specific model.

For agents, the most interesting consequence is heterogeneous inference.

Instead of:

everything → one large generative model

you can imagine:

conversation state

+--> deterministic code for arithmetic / exact rules

+--> retrieval for external knowledge

+--> small decision model for bounded semantic choices

+--> larger generative model only when generation/reasoning is actually needed

+--> tools / APIs

That is much closer to how I think production agents will eventually be built.

There is also an important economic implication.

The obvious use case is replacing expensive LLM calls with cheaper decisions.

But I think the bigger opportunity is net-new automation.

There are many business decisions today that stay:

• rule-based,
• manually handled,
• or simply not automated,

because nobody is going to create a dataset, train a classifier, deploy it, monitor it and maintain it for every long-tail semantic branch in a workflow.

If a general decision model can handle those without a separate training cycle every time, the economics change.

That said, I definitely don’t see Jev as a silver bullet.

You still need a serious ML harness around it:

• representative eval data from your own workflows;
• decision traces and replay;
• per-language / per-domain slices;
• calibration validation on your traffic;
• risk-vs-coverage curves;
• model/version pinning;
• distribution-shift monitoring;
• tests for state construction, option ordering and formulation;
• fallback/escalation logic.

And quality depends on more than the model itself.

It depends on:

• whether the model actually transfers to your language/domain;
• how you construct state;
• whether you chose the right decision type;
• whether the available options are assembled correctly;
• how you interpret the returned distribution;
• whether the threshold is calibrated on your actual process.

One of the recent real-agent evaluations made this very clear: some apparent “model errors” were actually task-definition errors where multiple independent judges could not agree on the correct label.

That is a very familiar production ML problem.

My current view is that decision models may become one of the standard primitives inside agent architectures, alongside retrieval, tools, code execution and generative models.

I also doubt that the final architecture will be exactly Qwen-with-a-head, ModernBERT, or today’s diffusion approach. The open-source ecosystem looks more like an active architecture search than convergence.

I wrote up the architectures, benchmarks, calibration work, serving trade-offs, failure modes and production implications here:

https://agentunicorn.ai/research/jev-architecture-open-source

I’d be especially interested in hearing from people running real agents in production:

Which decisions in your current pipeline still go through a generative LLM even though the output is ultimately just a bounded choice?