Choosing a Decision Model: Jev, Decider, d1-ultra, mammoth-1, and a Dozen More

🌏 中文版

Decision models replace an LLM's "generate some text and you write the parser" with "you list the options up front and the model returns a probability for each." After TypeSafe proposed System One on 2026-09-15, this category went from one vendor to more than a dozen by 2026-10-21: Jev 1.14, the OpenAI Decisions API, Perplexity Decider v1.2, Microsoft-Decision-1, Liquid d1-ultra, Base Labs' mammoth-1, vLLM's d3, StartLux-Decision, Fastino GLiDE, Celeris-1 Decision, Inception Mercury Decide, and Cloudflare Clef.

If what you build is a judgement over a finite set of options — which team gets this ticket, is this text spam, may this tool call run — this batch is worth a look. If you need knowledge or reasoning, the leaderboard numbers will tell you not to bother. What this piece gives you is the version matrix, what the two leaderboards that cannot be mixed actually measure, and the three places where things genuinely break once you wire one in.

What a decision model is: swapping "generate text" for "return probabilities"

When TypeSafe proposed System One models on 2026-09-15, the pitch was: structured program state goes in, typed probabilistic decisions come out. Jev was the first implementation, and the whole category ended up reusing the same interface.

A request looks like this — the System One API takes a state plus a set of questions, each tagged with the type of answer you want:

{
  "model": "jev-1.14.0",
  "state": "I was charged twice this month; please refund one of them.",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this ticket?",
      "criteria": { "billing": "payments and refunds", "technical": "product faults", "other": "none of the above" }
    },
    "refund": { "type": "noul", "instructions": "Is the customer asking for a refund?" },
    "urgency": {
      "type": "score",
      "instructions": "How urgent is it?",
      "criteria": ["can wait", "needs attention this week", "needed today"]
    }
  }
}

The response does not look like this:

{
  "answers": {
    "team": { "type": "choice", "choice": "billing", "probabilities": { "billing": 0.8, "technical": 0.15, "other": 0.05 }, "confidence": 0.7 },
    "refund": { "type": "noul", "noul": 0.97 },
    "urgency": { "type": "score", "score": 1.7, "confidence": 0.55, "legend": { "0": "can wait", "1": "needs attention this week", "2": "needed today" }, "probabilities": { "0": 0.1, "1": 0.1, "2": 0.8 } }
  },
  "usage": { "input_tokens": 120, "output_tokens": 12 }
}

Three points separate this from an LLM. First, the output is fixed fields, not free text — there is nothing to parse and no "type mismatch" failure mode; the schema you write and the type the model returns agree by construction. And output tokens are not billed: TypeSafe says it plainly, "You pay per input token. Output tokens are free." The OpenAI Decisions API likewise charges nothing for output or cache. (One common misconception deserves clearing up: that output tokens are zero. TypeSafe's own example reports usage.output_tokens as 12; only d3's example shows 0. The thing worth remembering is "not billed," not "zero.") Second, every option carries a score, so you can set a threshold: auto-approve above 0.9, hand off below it. Third, the three question types have distinct jobs: choice picks one of a finite set, noul is the probability that a statement is true, and score returns a probability-weighted position on an ordered scale (so the score can land between two levels).

But the confidence field means different things at different vendors, and that is the easiest trap in this category. At TypeSafe and Baseten, confidence is a normalized concentration: for choice the formula is (p_max − 1/n) / (1 − 1/n), so the three-way example above with p_max = 0.8 gives confidence = (0.8 − 1/3) / (1 − 1/3) = 0.7. noul has no confidence field at all. For score, confidence measures spread (a MAD-based quantity), not probability. vLLM's d3 uses a different convention entirely: there, confidence simply is the probability of the chosen option, and its noul answers carry an extra probabilities: {yes, no}. OpenAI returns a confidence field for choice and score but does not publish the formula.

So the practical conclusion is one line: threshold on probabilities[choice], not on confidence. Baseten's documentation says the same thing: "To apply a probability threshold to a choice, use probabilities[choice]."

Why now? Microsoft's framing is the most direct: a pipeline with twenty sequential decisions spends an extra two seconds for every extra 100 ms per decision. These "small decisions" used to require a frontier LLM, which meant paying to generate a paragraph of prose in exchange for a yes-or-no answer. Decision models cut that cost, and the price is that they only choose among a fixed set of options — they do not write reasons and they do not reason on their own.

The dozen-odd models, and what each one is

Two lists are more useful than one big table: the ones with weights you can host yourself, and the hosted-API-only ones. The latter cost a fixed service fee; the former cost GPU.

Open weights, self-hostable

Hosted API only

Prices are per million input tokens and output is not billed, which is the pricing structure the whole category shares: you are buying input tokens, not the words the model writes. The Perplexity model card also contains a sentence worth keeping in your selection notes: they trained on the official benchmark's training split (decontaminated against the test set) because "a model that did not would be compared unfairly." The vendor wrote it, and you should know what the comparison is actually measuring.

Two leaderboards, nearly opposite answers

Two different things are now called a decision model leaderboard. They measure different things and produce nearly opposite rankings. Their scores cannot be swapped, summed, or used to say who is stronger.

The community Jev Decision Index 0.4.1 — a community-maintained Space on Hugging Face tracking "what open reproductions of TypeSafe's Jev turn into." The 2026-10-22 05:40 UTC data snapshot holds 226 model rows (224 of them third-party reproductions; Jev 1.13 and 1.14 are the vendor's own reference rows), 42 benchmarks (37 of which make up the public panel), five areas, 110,201 requests (109,759 of them scoreable), all on a single RTX PRO 6000. The headline Full score is 60% private same-skill tests plus 40% private new-domain tasks, mapped back onto the public scale; public benchmarks are run but not scored, and the private tests are never published. This is a community-maintained Space — it is not official, and not Jev's own leaderboard.

Ranks use a 0.9-point tie band, so tied models share a rank (the site renders it as =3, =10):

Two things to take from this table. First, open reproductions as a whole now beat the original Jev — 40 of the 226 rows score higher than Jev 1.14, and the model at the top of the board, GLiDE V1.2, is a 27.8B full fine-tune whose base model is undisclosed and whose weights are not yet public (the model repository is private; the lab ran the weights the author shared). Second, the public subset and the Full score rank almost differently: mammoth-1's public subset of 75.34 is the highest on the board, while its Full score drops to =3. The "Decision Index 0.3 public subset 72.83 / 68.09" figures printed on vendor model cards are simply not the same thing as a 0.4.1 Full score — d3's card, for instance, claims "66.3, first on the official text board" on version 0.3.2, which was a 2026-10-12 snapshot and means nothing once the board version moves.

The Artificial Analysis Decision Index v1.0 — an independent third party, launched 2026-10-13, with four evaluations at 25% each: knowledge and hallucination (AA-Omniscience-Decision, half accuracy and half non-hallucination rate), reasoning (HLE-MultiChoice), tool selection (When2Call), and safety classification (WildGuardTest). Its population differs from the community board: it lists hosted-only models (gpt-6-luna, d1-ultra, Microsoft-Decision-1, Mercury Decide, Celeris-1 Decision) and Baseten-hosted Elephant 9B, but no StartLux, no d3, and no mammoth-1.

(Knowledge and non-hallucination are percentages; the rest are accuracies or F1. 16 decision models and 15 language models; scores from the 2026-10-22 board page, and boards move.)

The gap between the two boards comes down to the questions they ask: the community board asks "how well can you reproduce a model like Jev?"; Artificial Analysis asks "what is each model actually like at classification, tool selection, knowledge QA, and safety judgement?" The third number is the vendor's own report, and it is the one that ages fastest: in Base Labs' 2026-10-21 post they say mammoth-1 takes first place among 208 decision models on Jev Decision Index 0.4 (71.24), and scores 65.9 on the Artificial Analysis Decision Index with elephant-1 at 65.2 and tied for first with d1-ultra. Neither board agrees on 2026-10-22: the community board now has 226 rows with GLiDE V1.2 on top and mammoth-1 down at =3, and Artificial Analysis has no mammoth-1 row at all — Elephant 9B reads 64.69 and d1-ultra 65.25, not tied. The same model changes rank when you change the evaluation, so the only conclusion that travels is this: which evaluation you pick decides what you see.

Where the capability boundary sits: classification wins, knowledge loses, and everyone is overconfident

Split Artificial Analysis' four sub-evaluations apart and the shape is clear.

Classification and tool selection: decision models win. On safety classification (WildGuardTest), d1-ultra scores 95.4, GLiDE 95.1, Decider v1.2 94.4 — all above Claude Haiku 5.5 (Low) at 92.6 and GPT-6 Luna (Low) at 88.5. The gap on tool selection (When2Call) is larger: Elephant 9B 94.8, d1-ultra 92.6, Decider v1.2 87.7, against 72.6 for Haiku (Low) and 77.0 for Luna (Low). These two things are exactly what decision models were designed for.

Knowledge and reasoning: decision models lose. On the knowledge questions (AA-Omniscience-Decision) the decision models land between 11.6 and 41.0, against 52.4 for Claude Haiku 5.5 (Low), 66.0 for GPT-6 Luna (Low), and 93.4 for Claude Opus 5.5 (Max). On reasoning (HLE-MultiChoice) the best decision model is Jev 1.13 at 20.9, against 23.6 for Haiku (Low), 22.2 for Luna (Low), and 70.2 for Opus (Max). If you need knowledge or reasoning, use an LLM.

And they are broadly overconfident. This is Artificial Analysis' own conclusion: almost every decision model reports probabilities higher than its actual accuracy, concentrated in the knowledge and reasoning columns; tool selection and safety classification are comparatively well calibrated. The vendors know it too. Perplexity's model card says they calibrated with a temperature of 1.382 on held-out data, reaching ECE 0.014 and Brier 0.134 on their dev set, but an ECE of roughly 0.05 on decision data they had not seen — "the same for every model we measured." StartLux's card is blunter: a confidence value used to pick or route answers overstates its accuracy, so set thresholds on the answers your own policy returns, not on the model's confidence.

"I don't know" is the most important development in this category right now, but you have to read the column correctly first. AA defines it as: "the hallucination rate is the share of questions not answered correctly that were answered wrongly rather than with 'I don't know'," which means the non-hallucination rate measures how often, among the questions a model got wrong, it admitted ignorance instead of guessing. Convert that column and the knowledge column into absolute shares and you get three buckets — answered correctly, answered "I don't know," and confidently wrong. The following is this article's arithmetic, not a column AA publishes:

Several conclusions fall straight out of that table. Elephant 9B's high non-hallucination rate is not a skill; it is a model that gives up on 82.4% of knowledge questions — its accuracy is 11.6, the lowest on the board. Celeris-1 Decision's 0.0 means it never answers "I don't know" when it is wrong, which is the same as saying 62.2% of its knowledge questions are confidently wrong; d1 is the same story, 58%. And language models hedge too: Claude Haiku 5.5 (Low) says "I don't know" on 26% of questions while scoring 52.4 correct, while GPT-6 Luna (Low) never hedges and scores 66.

On launch day, the board read: "Microsoft-Decision-1 leads decision models at 40%, followed by Jev 1.13 at 35% and Mercury Decide at 32%. No other decision model reaches 17%." After Jev 1.14 (37.2) landed on 10-14, d1-ultra (84.5) on 10-15, and Elephant 9B (93.2) on 10-21, that sentence stopped being true. The real reason: some models started putting "I don't know" into the training objective. Base Labs' recipe is explicit about it: "I don't know" answers are part of the training objective, with a Brier calibration term on top of cross-entropy.

For you this is a hard selection constraint: if you set auto-approve thresholds off a confidence value, a model that admits ignorance is an order of magnitude safer than one that answers everything. But pick the model that admits ignorance and is still right — not the one that gives up on everything.

Cost and latency: what you actually save

Artificial Analysis converts "the average cost of running its own evaluation suite per task" into dollars per million tasks. That is not the cost of your workflow, but comparing models against each other inside the same table is valid.

Three conclusions you can use directly. First, the order of magnitude: the cheapest decision models (Decider Flash v1.2 and Elephant 9B, $8 each) are 11× cheaper than the cheapest language-model comparator (GPT-6 Luna Non-reasoning, $88) and more than six thousand times cheaper than Opus (Max). Second, "decision models are always cheaper" is false: the most expensive decision model, Clef, costs $199 per million tasks — more than Claude Haiku 5.5 (Low) at $184 and more than GPT-6 Luna (Low) at $131 — and the reason is list price: Clef's $0.24/1M input is the priciest input rate on the board, while Decider Flash's $0.01/1M puts it at the bottom. Third, latency is a clean sweep: decision models run 63–500 ms, language models 1,136–14,388 ms.

But those two columns are not the same yardstick, and AA says so: "Decision models are billed for input tokens only, and language models for input and output tokens." The language-model rows include the cost of generating the prose, so they look more expensive than "the same decision" would.

Self-hosting costs show up elsewhere. d3's card reports 65.4 GiB of memory per replica on an MI325X. StartLux needs one H200 for 27B BF16, but a 4-bit MLX build fits in 64 GB on an M5 Pro — at the cost of a 1,024-token prompt going from 98 ms on H200 (vLLM BF16) to 3,308 ms on the M5 Pro (MLX 4-bit), roughly 34×, and MLX's BF16 does not fit in 64 GB at all. d1-3B answers a single question in 16 ms on a Jetson AGX Thor and three questions in 20 ms, or 8 ms for one question on an RTX 4090. If your volume is "a few hundred calls a minute," a hosted API is usually still cheaper than one GPU; if it is "a few hundred a second" or the data cannot leave the machine, self-hosting wins.

Wiring it into an agent: three implementation points

Here is the overall routing shape first. The standard way to use a decision model inside an agent is: send the state and the typed questions, get back a probability distribution per question, and then let your own policy decide whether to auto-approve:

One: ask every question in a single call, but do not put competing options in the same request — and that is the 27B's specialty, not a general rule. Perplexity's joint request on the 27B is the most valuable engineering idea in this batch: all questions share one sequence, each is read out at a marker token, so asking three questions costs about as much as asking one, and the questions can see each other. The official docs give you the criterion for when to group: different facets of the same decision should be merged (officially measured SATA-Bench +11.3, ACOS +7.8, PhishNChips +5.1), while competing candidates should be sent separately (BFCL −1.1, BRIGHT −2.9, ToolRet −4.8). Put "how much should this order be refunded" and "should this order be refunded at all" in the same request and the model will contaminate one with the other. But the 4B Flash is the other way round: the docs say it is "more accurate with one question per request" on most multi-question tasks, its public scores agree (60.05 one-question-per-request against 58.88 joint), and multiple questions make Flash re-bill the state (three questions against a 10,000-token state cost roughly 30,000 input tokens). Send Flash one question at a time.

Two: the threshold and the human review are your policy, not the model's. The model only produces scores; "what counts as good enough to auto-approve" is an application decision. Baseten's docs give the pattern directly: option probability ≥ 0.9 means dispatch, otherwise hand it to a human.

curl -s https://inference.baseten.co/v1/systemone -H 'content-type: application/json' -d '{
  "model": "decision-models/baseten-elephant-9b-latest",
  "state": {"ticket": "I was charged twice this month; please refund one of them."},
  "questions": {
    "team": {"type": "choice", "instructions": "Which team should handle this ticket?",
             "criteria": {"billing": "payments and refunds", "technical": "product faults"}}
  }
}' | jq -r '
  .answers.team |
  if .probabilities[.choice] >= 0.9 then .choice else "manual_review" end
'

Note that jq keys off probabilities[.choice] and not confidence — that is the point made above.

This snippet is a rewrite of Baseten's documentation (I swapped in an English ticket and the hosted endpoint); this site has not actually deployed that request.

There is also a versioning trap: Jev's jev-latest currently resolves to 1.13.0, and jev-preview is the one on 1.14.0, and the docs warn that "an alias moves when a new release ships, so its answers can change without any change to your code" — if you tuned a threshold against a particular version, send that version's ID rather than the alias.

Three: switching vendors is cheap, except for OpenAI. Most vendors take the System One schema (state plus questions, with type set to choice / noul / score), so swapping models usually means changing the model field. OpenAI is the exception: it calls the type predicate, the endpoint is /v1/decisions, questions is an array where every entry needs a name, the state field is input, options are a choices array of {value, description} rather than a criteria map, and there is a refusal response type. If you want several vendors in one codebase, an adapter layer is the cheapest path.

Three capacity limits worth remembering: StartLux covers at most 26 options per read and groups beyond that; Perplexity allows 255; Jev allows 64k per request and 32k per question; and Perplexity's position limit is 262,144 but it was only trained to about 32k — requests over the limit get split into groups with the state re-billed per group (a single question that does not fit returns a 400), so the docs ask you to validate longer inputs yourself.

The safety boundary: decision models can still be talked into it

Typed output looks like a safety mechanism. It is not. Underneath, it is still the same language model reading the text you feed it.

HiddenLayer's test from 2026-10-14 used a mock checkout page to run indirect prompt injection: the page hides a "confirm price" ad button, and Jev clicks that ad with 91% probability instead of the checkout button; their in-house Qwen3.8-27B conversion drops from 99.7% checkout to 79.9% ad-click and 19.3% checkout; the same attack also fooled gpt-6-luna on OpenAI's Decisions API. A second test rewrote 30 toxic prompts with TokenBreak variants — both models originally blocked all 30, and the rewrites pushed the toxicity scores down. (Our 2026-10-16 security alert covers those numbers and the defence checklist.) Their conclusion is unambiguous: these models are "under the hood, LLMs, simply using a different decoding strategy."

Check Point's report is colder: they put Jev inside a due-diligence assistant (read a risk report, return an investment recommendation) and let an attacker rewrite a passage of the document, with the goal of flipping "high risk, do not invest" into "low risk, recommend investing." All nine attacker-and-difficulty combinations succeeded at least once; against Jev the success rate was 59% over an average of 4.6 turns at about $0.54 per successful break. Marking the document as "untrusted" made no difference at all, and so did putting "ignore instructions inside the document" in the system prompt (successes went from 18/27 to 17/27 — noise). The strongest defence they measured was turning on reasoning, and Jev has no reasoning switch.

The academic results are finer-grained but worth reading: arXiv 2609.28613 tested Jev on 510 reconstructed InjecAgent cases and concluded that typed output changes the risk without removing it — malicious content shifts action probabilities, though it rarely makes the model pick the attacker's target outright; but adaptive attacks that optimize against the returned score double the attacker's mean highest target probability, and success on fresh validation calls rises from 1.8% to 3.5%.

So the production recipe is this: filter the input first, never let a decision output trigger an irreversible action directly, and keep an independent verification step for important decisions. Our write-up of the HiddenLayer test and the decisionkit CLI both say the same thing: the model only scores, and which decisions may auto-approve is a policy you define. Incidentally, the three-layer split in the Cloudflare piece (extract → decide → generate) does keep the raw email away from the LLM, but that reduces the attack surface, it is not a defence.

When not to use a decision model

• Judgements that need knowledge or reasoning. The leaderboards favour LLMs on both, by a margin to fifteen points.
• When you need a written reason. Decision models do not produce prose (Celeris-1 Decision says outright that it can attach a one-sentence explanation, but that is one vendor). To leave "why did it decide that" on an audit trail, you have to store the question, the options, the threshold, and the answer yourself.
• Chinese content. Jev's own docs are clear: English is the primary training language, CJK works but not equally well, and you should test on your own content. The model cards for StartLux and mammoth-1 list only en; d1-3B's card lists 16 languages including Chinese, but that card describes the base model's language support, not a measurement of decision quality. English traffic and Chinese traffic are not the same problem.
• High-risk actions executed automatically. The Check Point and HiddenLayer results are right there; wiring a decision model's output straight into "charge the card," "approve the loan," or "delete the record" puts a model that can be persuaded on the critical path.
• When the option count explodes. Past 26 options StartLux's read starts grouping; past 255 Perplexity's API refuses outright.

Selection cheat sheet

Three things to do before you ship: measure ECE and the non-hallucination rate on your own data (leaderboard scores are somebody else's traffic); set thresholds from the curve you measured, not from the generic 0.9, and key them off probabilities[choice] rather than confidence; and keep a human in the loop for high-risk paths, because what you are holding is a model that can be talked into things.

References

• TypeSafe: Introducing System One Models & Jev (2026-09-15)
• TypeSafe Docs: Models (Jev 1.14, context, the long-context header, aliases, language support)
• TypeSafe Docs: Confidence (how each question type defines confidence)
• OpenAI Docs: Decisions API (question types, pricing, threshold guidance, residency and ZDR)
• OpenAI Changelog: v1/decisions public beta on 2026-10-06
• Perplexity Docs: Changelog (Decider v1.2 at $0.02, Flash v1.2 at $0.01)
• Hugging Face: pplx-decider-v1.2 model card (joint requests, calibration, training-data disclosure)
• Hugging Face: pplx-decider-flash-v1.2 model card (distillation, one-question-per-request, Decision Index 0.3 public scores)
• Microsoft: Introducing Microsoft-Decision-1 (2026-10-09)
• OpenRouter: microsoft/microsoft-decision-1 (the alpha decisions endpoint)
• Liquid AI: d1-ultra (2026-10-15)
• Hugging Face Blog: Multimodal open d1 decision models for the edge (2026-10-07)
• Liquid AI Docs: Decision Models (question types, image billing, when to use an LLM)
• Base Labs / Baseten: mammoth-1 model card (training recipe, Brier calibration, the "I don't know" option)
• Base Labs: The top ranked decision models in the world (2026-10-21)
• Baseten Docs: Decision models (the threshold-routing example)
• vLLM Semantic Router: d3 model card (video input, vLLM support, speed, memory)
• vLLM v0.32.0 release notes (the /v1/systemone endpoint)
• StartLux-Decision v1.1 model card (Instant / AutoThink, license, thresholds, latency table)
• arXiv 2610.17309: StartLux-Decision technical report
• Fastino: Introducing GLiDE (2026-09-30)
• Fastino Docs: GLiDE (pricing, context)
• Celeris: Pricing (Celeris-1 Decision)
• Inception Docs: Models (Mercury Decide, the v1/decisions endpoint)
• Inception Docs: Decisions (the extract question type and its limits)
• Strands Decider 2B model card (Apache-2.0, public training data)
• Hugging Face Space: Jev Decision Index (the 0.4.1 community board and its data)
• Artificial Analysis: Decision Index v1.0 (2026-10-13, method and conclusions)
• Artificial Analysis: Decision Index board page (2026-10-22 scores, cost and latency definitions)
• Artificial Analysis: Decision Index methodology (the non-hallucination rate definition)
• HiddenLayer: Jevil in the details (2026-10-14)
• Check Point: A Decision Model Breaks Like Any Other Language Model (2026-09-24)
• arXiv 2609.28613: Decision Hijacking (the injection risk of typed output)
• Cloudflare: Introducing Clef (2026-10-01)
• This site: Cloudflare Model Family: Choosing Clef Decision Models and Clairic Extraction Models
• This site: Security alert: decision model Jev is not immune to prompt injection
• This site: Tool recommendation: decisionkit
• This site: Model card: Jev (TypeSafe AI)