🌏 中文版
Flide (paper: arXiv:2610.42148, authors from the National University of Singapore and the University of Oxford) is an inference engine built for decision models. A decision model reads a long context, answers several multiple-choice questions about it, and outputs only a probability for each option, with no generated text. Flide computes the shared context once, then forks it into each question, and every answer matches what the model gives when asked that question alone.
The authors claim up to 9.5x faster than vLLM and SGLang, but that is the most favorable setup: the smallest model, with one long article and 18 questions. Before deciding whether to switch, look at the shape of your requests. "One input, many questions" gets the large gap; "one question, many inputs" gets little today.
What a decision model is and who Flide serves
A decision model takes a state (a conversation, an agent trajectory, an email) and outputs probabilities over options defined in advance for each question. Typical uses are routing requests, scoring answers, and choosing an agent's next step, with the calling program applying a probability threshold such as 0.5.
TypeSafe AI introduced the category in September 2026 with Jev, currently a closed, early-access service. The speed and cost figures in that post are the vendor's own, and its benchmark uses the average of large models as the reference answer. Flide serves open models compatible with Jev's /v1/systemone API: Open-Jev, OneJev (built by the Flide authors' own team), and general chat models such as Qwen3.5 and Gemma 4.
You need to know what a KV cache and prefix caching are to follow the rest.
Where current engines waste work
The paper traces the problem to one mismatch: logically a request is a tree (the shared state is the trunk, each question a branch), but current engines do not execute it that way.
Recomputation. vLLM and SGLang treat each question as an independent prompt. vLLM has prefix caching, but hybrid models like Qwen3.5 replace most attention layers with Gated DeltaNet recurrent layers, which compress all earlier tokens into one fixed-size state. vLLM's documentation says the cache can only resume at a block boundary, so every question recomputes the tokens past the last boundary.
Shifted answers. SGLang's multi-item scoring appends all questions to one shared context with a delimiter token before each, which does compute the context once. The delimiter changes each question's hidden states and shifts the probabilities: on Qwen3.5-0.8B the paper measures up to 28.9% of decisions flipping (landing on the other side of the threshold), against at most 2.3% for Flide.
Floating-point summation order also changes with batch size, so the same request can get different answers under different load. Comparing light and heavy load, up to 3.2% of answers change on vLLM, against at most 0.2% on Flide's default mode.
Simplified from the paper's Figures 1 and 2; tag 0 is the trunk.
How Flide does it
One stream, with the tree kept as GPU data. Flide flattens a round of requests into a token sequence of the form [state | question 1 | question 2 ...], and each token carries three indices: request, branch, and position. Question positions continue from the end of the state, so the rotary embedding matches what the question would get when asked alone.
ForkAttention uses a different trick in each layer type. Full-attention layers use a visibility rule: a token sees only its own request's trunk and its own branch, and nothing in the future. Gated DeltaNet layers have no attention matrix to mask, so each layer runs two calls: the first processes every request's state and saves the recurrent state at its end; the second processes all questions at once, and each question starts from its own request's saved state through an index, without copying anything. The paper proves that in exact arithmetic every question's hidden state equals what you get processing state + question in isolation; in floating point the difference stays at rounding level.
A few CUDA graphs serve every request mix. Request lengths and question counts live in GPU buffers, and CUDA graphs are indexed only by stream length, so no graph is needed per combination. A decision request takes just a few milliseconds of GPU time, so the saved kernel-launch overhead is why Flide is faster even with one question per state.
Agent histories can resume. At each step Flide saves a checkpoint slightly before the end of the state, and the next step computes only the new tokens and a short tail. On SWE-agent trajectories, each extra 1k tokens of history adds about 1 millisecond per step.
Bitwise determinism is optional. With --batch-invariant, a request returns the same bits under any load, at a cost of at most 10% throughput at 64 clients.
A concrete example: ten thousand emails, eight questions each
The README's email-triage demo asks eight questions of each of 10,000 emails and tags them, using OneJev on one H200: the whole batch runs 4.1x faster than the fastest other engine, and a single email 3.5x faster.
The call looks like this (from the README):
uv venv -p 3.12 && source .venv/bin/activate
uv pip install flide torchvision
flide serve --model OmniJev/OneJev1.6-0.8B
curl http://localhost:8000/v1/systemone -H "Content-Type: application/json" -d '{
"state": "Order #1182 arrived with a cracked screen. The customer asks for a refund and attached two photos.",
"questions": {
"action": {"type": "choice", "instructions": "What should support do next?",
"criteria": {"refund": "refund the order", "replace": "send a replacement", "escalate": "ask a human"}},
"urgent": {"type": "noul", "instructions": "Does this need a reply within the hour?"}
}
}'
Following this example, put everything shared into state and send all questions about one input in a single request. For continuous agent decisions, add --session-cache-gb 40 so the next step reads only what is new.
Measured: how much faster in which setting
The comparison in the table below is against the faster of the two baselines in each setting (vLLM or SGLang unless noted), measured in the paper on a single H200.
| Setting | Flide's gap | Conditions |
|---|---|---|
| One long article, 18 questions | 9.5x lower latency, 8.5x throughput | Qwen3.5-0.8B, single client, the best cell |
| One 2k-token state, 64 questions | 12.7x faster (vs vLLM) | Qwen3.5-4B |
| Shared-state workloads, averaged over four model sizes | about 1.3x to 3.3x | Ratios of the averages row in the paper's Table 3; varies by model family and dataset |
| One question per state | about 1x to 3x | Largest for small models on short inputs; near tie for 27B on long inputs |
| An image plus 11 questions | 5.0x to 7.5x | GQA, Qwen3.5 0.8B to 9B |
| Continuous agent decisions | 4.4x to 7.2x steps per second | SWE-agent trajectories |
| Against vllm-jev, the engine written for Open-Jev | about 3x on short inputs, up to 1.7x on long | Open-Jev-2B and 9B |
On accuracy, Flide stays within 1.2 percentage points of vLLM, and its answers are closest to the float32 reference in total variation distance among the engines compared.
Contemporaneous related work
Flide's related-work section cites several contemporaneous papers. I read their full texts. None of them cites Flide, and all were submitted earlier, so each solves a different problem and the comparison needs care.
- The encoder line: Verdict (arXiv:2610.20778) and Narrow Decision Models (arXiv:2610.25553). Both use a ModernBERT-style encoder with a block attention mask that answers every question about one state in a single forward pass, and both require training the model that way. They do not serve off-the-shelf hybrid decoder models, so they are not Flide's direct competitors. Flide describes this family as paying a latency penalty for scanning block masks, but neither paper measures the mask's own cost; Verdict measures the opposite direction: with the same weights, packing is 2.7x to 2.8x faster than running each question separately (batch 8, one laptop GPU).
- Numerical guarantees stop at rounding level there too. Both say the invariances hold in exact arithmetic and report small measured error; in bf16, Verdict finds that asking one question alone versus packed moves 0.75% of cases across the auto-handling threshold. That matches Flide's own statement; what Flide adds is the optional
--batch-invariant. - FastJEV (arXiv:2610.11379). Also a Gated DeltaNet hybrid, but what it shares is the candidate answers of a single question, not multiple questions. The authors' own conclusion is that tree sharing is often slower in practice: the full method is slightly slower than the original, and the fast variant is the one without the tree. That supports Flide's choice to flatten branches into one stream and avoid segmented passes, though FastJEV does not compare against Flide.
- Prefix caching on hybrids: RadixFlow (arXiv:2610.18143) and SuffixReplay (arXiv:2609.33477, not cited by Flide). RadixFlow takes "resuming needs the full state at the boundary" as a premise and addresses scheduling: idle slots when a consumer waits on a producer in the same round. SuffixReplay stores sparse hidden states and, on a hit, replays a recent suffix to approximately restore the state at every 64-token page boundary; it also notes that SGLang by default saves checkpoints at request end and at the first branching continuation, not only at chunk boundaries. So "resume only from block boundaries" holds for vLLM's documentation, not as a rule for every engine. SuffixReplay is approximate, whereas Flide's selling point is exactness.
None of these papers was compared against Flide on Flide's workloads, so the above marks the edges of the problem; it is not evidence about who is faster.
Limits and where it does not help
- All numbers come from the authors' own tests: a single H200, a pre-release vLLM, SGLang at default settings, and no third-party reproduction yet. Before relying on it, run the
flide verifycommand mentioned in the README on your own model and requests. - One question over many inputs gets little. A Hacker News commenter pointed out this shape is more common in practice; the author replied that the current setup does not help much there and that a future version might put the question first so all states share it.
- Bidirectional-attention models speed up less. Models that read the whole prompt both ways, even for the question (the paper's Appendix H lists Jeffan, StartLux, and Lychee), run each question as its own sequence inside the stream. The author said on HN that GLiDE uses bidirectional attention and is not yet supported.
- The gap is smaller on full-attention models. On Gemma 4, Flide is 1.1x to 2.9x faster than vLLM, which already caches the shared state. A Hacker News commenter said they modified vLLM and got similar results on such models, without sharing numbers.
- The gap over SGLang depends on its checkpoint settings. According to the SuffixReplay paper, SGLang by default also saves checkpoints at request end and at the first branching continuation, so "every question recomputes the tail" is the worst case. Flide's SGLang numbers were measured at default settings, and I found no comparison varying the checkpoint settings.
- Exactness holds only to rounding level in floating point. Bit-identical output needs
--batch-invariant. - Platform. Prebuilt kernels cover the A40, A10, H100, H200, and the RTX 30 through 50 series. The authors tested only on Linux; on Windows you need WSL2.
In the end
Flide's trade is simple: it hands the "one state, many questions" structure of decision tasks to the engine, in exchange for computing the shared state once while keeping each answer identical to asking alone. If your workload asks several questions about one email, conversation, or agent trajectory, it is worth running on your own data; if it asks one question about many different inputs, or the model reads bidirectionally, the gain is small today.
The decision-model category is still new: Jev's performance claims come from the vendor, and Flide's speed claims come from its authors. Both have public code to try, but independent third-party numbers are still to come.
References
- Flide: Accelerating Decision Inference with Exact State Forking (arXiv:2610.42148)
- OmniJev/Flide (GitHub, Apache-2.0)
- Flide project page
- Introducing System One Models & Jev (TypeSafe AI)
- Automatic Prefix Caching (vLLM documentation)
- Show HN: Flide (Hacker News thread)
- Verdict: Encode Once, Decide Many (arXiv:2610.20778)
- Narrow Decision Models (arXiv:2610.25553)
- FastJEV (arXiv:2610.11379)
- RadixFlow (arXiv:2610.18143)
- Prefix Caching via SuffixReplay for Hybrid LLMs (arXiv:2609.33477)
- Related on this site: vLLM: From PagedAttention to a Production LLM Inference Engine
Update log
- 2026-11-13: Read the full texts of the contemporaneous papers Flide cites; added "Contemporaneous related work" and one limit, plus five references.