Jev vs GPT-6 vs Claude vs the Open Clones: We Benchmarked 150 Business Decisions

Early in the weekend we ran a quick test of Jev, the typed-decision model TypeSafe AI put into early access on 15 September, against the open reproduction everyone was pointing at. Fifteen records out of our own CRM. Jev scored 15/15, the open model scored 4/15, we rewrote the criteria, the open model went to 14/15, and we wrote down the tidy conclusion: the open weights are nearly as good, they are just less forgiving.

That conclusion did not survive a real test. Before the weekend was out we had built one — 150 hand-labelled records across three jobs a business actually runs, seven models, four different wordings of every question — and the open model landed at 61%. Here is what held up and what did not.

What a System One model is

A System One model answers typed questions in a single forward pass and generates no text. You hand it unstructured state and a schema of questions; it returns a label from a fixed set, a probability, or a position on a scale, each with a calibrated confidence. It cannot hallucinate a category that was not in the schema, because the valid answers are declared in advance.

There are three question types. A choice picks one of up to 255 options. A noul returns a probability from 0.0 to 1.0 for a yes/no statement. A score returns a position on an ordered scale. Ask four questions in one call and they all resolve in the same pass, so four cost roughly what one does.

It is not a smaller language model. It is an if-statement that can read.

The fastest way to see what that means is to give it a job with a visible answer. We wired Jev to a game of Snake: every frame, the board goes in as a sentence of text and one typed choice comes back — up, down, left or right, with a probability on each. No reasoning, no tokens, no prompt engineering. It played 260 frames, ate 21, and never hit a wall.

Watch the bars rather than the snake. The model is not planning a route; it is answering the same narrow question 260 times in a row, in 408 ms each, and the distribution collapses onto one option the moment the board makes the answer obvious. That is the whole product.

How we tested it

Three jobs, fifty records each, every label written by hand before any model ran: inbound lead triage for a consultancy inbox, support ticket routing for a B2B SaaS desk, and the one we care most about — classifying the risk of a shell command an autonomous agent wants to run, across read-only, reversible write, destructive, and exfiltration. The records are written rather than pulled, so that a public post can show the exact data; they are modelled on the real streams and deliberately include the boring middle, broken metadata, and the cases built to tempt a wrong answer.

The run itself was five steps, and the order of them is the method:

• Label first, before any model runs. Every record was read and classified by hand up front. Labelling from model output is how a benchmark quietly measures agreement instead of accuracy.
• Flatten each record to one state string. A System One model takes unstructured state, not a chat transcript. The same string goes to every model, so nothing differs between them but the model.
• Ask every question in one pass. One choice, one or two nouls, one score per record — they resolve together, so four questions cost about what one does. The language models got the identical criteria text as a prompt and were asked for JSON.
• Sweep the wording four ways. A terse one-liner per option; a rich version with three or four sentences and examples; that plus explicit defaults and exclusions; and a deliberately lazy four-or-five words, as a floor test. Without this you cannot tell a bad model from a badly written question.
• Score accuracy, calibration and latency separately. Choice accuracy against the labels, noul thresholds swept from 0.3 to 0.95 rather than assumed, and the confidence gate scored on both halves — errors caught and volume escalated.

Fifty-seven runs in all, against five language models — GPT-6 Astra, GPT-5.2, GPT-5-mini, GPT-4.1-mini and Claude Haiku 4.5 — plus Jev 1.13.0 and two open models.

Why those two open models

There are around twenty open reproductions of the System One shape now, and picking two needed a rule. The rule we used: TypeSafe has never described RLCD, its calibration training method, in enough detail to copy, so no open project reproduces it — most approximate it with post-hoc temperature scaling and several say plainly in their own READMEs that their probabilities are not estimates of correctness. Only two publish calibration work at all, so those are the two we ran.

• Von — 395M parameters on ModernBERT-large, Apache-2.0, installs as pip install von-sdk and runs in-process on CPU or Apple Silicon. Post- trained with a combined cross-entropy and Brier loss and shipped with a fitted temperature, which is the closest thing to Jev's calibration story in the open.
• Laya — 421M, also ModernBERT-large, Apache-2.0, with a decision head trained from scratch and a published expected-calibration-error figure. Multilingual, and the only one of the two with an ONNX path for Node.

Both answer the same three question types through the same response shape as Jev, so the swap is a few lines of code. That is exactly why the accuracy gap matters: nothing about the integration warns you.

Result one: Jev matches the frontier models on accuracy and costs 212 times less

Read that list top to bottom and the ordering is the finding. The newest and most expensive model in the test, GPT-6 Astra at $10 per million input tokens, is two records worse than GPT-5.2 and one worse than GPT-4.1-mini, which costs twenty-five times less. These tasks are saturated. Past some modest capability bar every competent model answers them the same way, and everything you pay above that bar buys reasoning the task never asked for.

Jev ties the best language model in the test and costs 212 times less than the newest one. Haiku is the only hosted model that separated from the pack, and it did so on ticket routing (43/50), the task with the most genuinely ambiguous records. We measured it through the Claude Code CLI, which runs it with extended thinking on, and 95% of its output tokens were reasoning it then discarded — hence the cost range: $3.89 per thousand as run, $0.68 if you strip the thinking. Either figure is many times Jev for the same answer.

Jev made five errors in 150, and the two we inspected closely are the shape of the failure. One was an S3 sync to a local folder: we labelled it read-only because nothing leaves the machine, Jev called it a reversible write because files land on disk, and Jev is arguably right — it returned that answer at 0.45 confidence. The one it got confidently wrong was SEO backlink spam, which we called noise and it called a vendor pitch at confidence 1.0.

Result two: the speed headline is real, and it is not measured against what you think

This is the claim that travelled furthest in launch week, and it is the one we most wanted a number for:

Median wall-clock latency per record, measured from a US connection:

• Laya, local — 102 ms
• Von, local — 196 ms
• Jev, hosted — 423 ms
• GPT-4.1-mini — 701 ms
• GPT-5.2 — 1,000 ms
• GPT-6 Astra — 2,508 ms
• Claude Haiku 4.5, measured through the Claude Code CLI — 6,835 ms

The Haiku figure is not a like-for-like API latency: it carries CLI start-up and the extended thinking that produced 95% of its output tokens. Treat GPT-4.1-mini at 701 ms as the fair language-model reference.

TypeSafe publishes a 40x-200x speedup, and at 423 ms Jev sits comfortably inside the 70-500 ms band it quotes for itself. Read the other half of that ratio, though: the baseline is the 3 to 329 seconds a frontier model takes on a System One workflow. That is a multi-step agentic run, not one classification call. Against a single well-scoped call to a small language model, Jev is 1.7 times faster end to end, because a network round trip costs what a network round trip costs. If you want the order-of-magnitude win, you have to run weights locally — which is the one thing Jev does not offer, and the one thing Von and Laya do.

Result three: the open reproductions are not drop-in replacements

This is the finding that overturned the first pass. On the agent guardrail task, Von scored 23/50 and Laya 21/50 against Jev's 49/50. They do not degrade gracefully — they collapse onto one class. Von predicted exfiltration 25 times in 50 records on a set that contains five. In a production guardrail that is a leak alarm on ls -la.

The obvious mitigation is a confidence gate: escalate anything the small model is unsure about to Claude. We scored both halves of that gate, which is the only honest way to read one. At a 0.9 threshold on the command task, Jev caught its single error while escalating 8% of volume. Von caught 26 of its 27 errors while escalating 92% of volume. Laya escalated 100%.

A gate that catches every error by escalating every record is not a gate. It is a slow path with extra steps.

Result four: wording moves every model, and the floor is the only number that matters

We expected Jev to be the wording-proof one. It is not. Running all four wordings of the guardrail question through six models, worst config to best:

Jev swings ten points across the four wordings. So does GPT-4.1-mini. Haiku swings four and is the steadiest model in the test. Whatever advantage calibration training confers, insensitivity to how you phrase the question is not it, and anyone selling that should be asked for the floor.

The floor is the whole story. Every hosted model bottoms out somewhere between 82 and 92% — bad wording costs you points but leaves you with a working feature. The open reproductions bottom out at 18 and 36%, which is not a feature at all. And richer criteria do not reliably buy the difference back: on lead triage Von scored 34/50 on the terse one-line wording and 28/50 on the carefully written one. What moves those numbers is sensitivity to surface form, not comprehension, so every future edit to your criteria is an unannounced regression risk and the only thing standing between you and a silent one is a labelled eval set you maintain forever.

Result five: never ship a probability threshold of 0.5

Jev's noul has a floor. On records that plainly needed no human, it still returned 0.2 to 0.5. Thresholding its "does this need approval" probability at 0.5 gave 80% accuracy on the command task; sweeping the threshold and cutting at 0.7 gave 92%. Same model, same answers, twelve points from one constant. Thresholds do not transfer between models either — we had to re-sweep for every one.

What this actually means

The interesting result is not that a hosted model beats open weights. It is that the category is real and the cost curve underneath it is about to reprice a specific kind of work. Most production LLM calls return a label, not prose — a route, a score, a yes/no, a category — and every one of them is paying for autoregressive generation it parses away. Jev is the first model we have tested that does that job at frontier accuracy without the generation.

That last clause is the part with teeth. The JSON prompting, the parse, the retry when the parse fails, the schema validation, the fallback when validation fails — all of it is scaffolding around the fact that a language model returns a string. A typed decision deletes the scaffolding, not just the tokens.

Jev does not replace Claude. It is what makes keeping Claude affordable

The framing everyone reaches for is Jev versus an LLM, and it is the wrong one. They answer different halves of a problem. A typed model decides; a language model explains, writes, and handles the case nobody specified in advance. A decision has no words in it, and everything downstream of a decision does.

The pattern that works is both, in series. Put the cheap typed model in front of the expensive one and let it carry the volume:

• Jev sees every record. Every inbound message, every tool call, every ticket — at 400 ms and four cents per thousand, you can afford to look at all of them.
• Claude sees the ones that need words. The escalations, the low confidence answers, the replies a human will read, the judgment calls the schema did not anticipate.
• The gate is the confidence number, calibrated against a labelled set rather than guessed. On our command task that routed 8% of volume to the slow path and caught the one error in the other 92%.

The cheap model is not there to replace the expensive one. It is there to make it affordable to keep the expensive one in the loop at all.

This is why the guardrail case is the one we care about most. Nobody can afford a frontier call before every shell command an agent runs, so in practice teams either skip the check or run a regex. A 400 ms typed check makes the safety layer cheap enough to actually exist, and Claude still gets the three commands an hour that are genuinely ambiguous.

Which one to use, and when to use both

Three cases, and most teams have all three running at once.

Use the typed model alone

The answer set is known, nobody downstream reads a justification, and the decision runs constantly or a person is waiting on it. In the work we see, that is inbound lead routing in front of the CRM write rather than in a nightly batch; support triage assigning department and severity on arrival so the queue is ordered before a human opens it; agent guardrails classifying whether a tool call is read-only, reversible or destructive; invoice and document exception flagging, where 95% match the template and the job is finding the 5% that do not; model routing, where the routing decision costs a fraction of the call it routes; and moderation or intake screening, where every submission is judged before it appears.

Use the language model alone

Anything that produces words a person will read, and anything whose answer set you cannot write down in advance. Drafting the reply to the ticket, summarising the call, writing the proposal, explaining to an auditor why a transaction was flagged, answering a question nobody anticipated. Also — and this is the one teams get wrong in the other direction — anything low volume. A nightly cron over fifty records does not care that one option costs $0.30 per thousand and the other $0.04. Moving it buys you a vendor, an integration, and an eval set to maintain, in exchange for nothing.

Combine them, which is the common case

Put the typed model in front and let it carry the volume, with the language model on the escalations. Support triage is the clearest example: Jev assigns department, severity and churn risk on every ticket as it lands, and Claude drafts the reply for the ones a human is about to send. Lead routing is the same shape — typed classification on every inbound, Claude writing the first-touch email only for the ones that qualify. So is the agent loop: a typed check before every tool call, and Claude asked to reason about the handful the check flags.

The seam is the confidence number, calibrated against a labelled set rather than guessed. On our command task a 0.9 gate routed 8% of volume to the slow path and caught the one error in the other 92%. Get that gate right and the economics follow: on a guardrail firing on every tool call in an agent loop at ten million decisions a month, it is $2,960 on GPT-4.1-mini, $76,420 on GPT-6 Astra, and $360 on Jev. That gap is the whole argument.

Where we would use it, we would ship it behind a flag with the Claude call still in the code path and both answers logged for a week before cutting over. It is a single vendor, five days old, in early access, with no SLA. We would not put a client-facing critical path on it without a fallback — and we would not run it anywhere without the eval set, which is the only reason we know our own first conclusion was wrong.

What we are not claiming

The 150 records are synthetic — realistic in shape and deliberately messy, but written by us, which makes them cleaner than a real inbox and almost certainly easier than production. Fifty records per task is enough to separate 97% from 61% and nowhere near enough to separate 97% from 96%; treat the top five hosted models as tied. We tested English text only, two of the seven models come from the same lab, and every number here comes from one weekend on one machine and one connection. The eval harness and the labelled data are ours to share — ask and we will send them.

If you want to know whether a decision in your own stack is one of these, the test is cheap: find the LLM calls whose output you parse rather than display, count how often they run, and label two hundred of them by hand before you believe any model about any of it. That last step is the whole exercise. We have published the skills we use to run it, and it is the same discipline behind our model comparisons. If you would rather we ran the audit against your own traffic, that is a normal first engagement for our agent and Claude Code work.

Sources

• Introducing System One Models and Jev - TypeSafe AI
• Models and pricing ($0.042 per million input tokens, output free) - TypeSafe AI Docs
• System One concepts and the calibration caveat - TypeSafe AI Docs
• Von, the open-source System One decision model - GitHub
• Laya, ModernBERT-large with RLCD decision heads - Hugging Face
• Open-source Jev alternatives and reproductions - System One Models
• Model pricing used for the cost column - OpenRouter