Bobo Li and colleagues at the National University of Singapore and Oxford published a preprint on 30 October describing Flide, an inference engine that runs decision-model queries up to 9.5 times faster than vLLM and SGLang S¹P². That speedup targets a gap the authors say current engines leave wide open: generation-oriented systems either recompute shared prefixes repeatedly or distort the output probabilities that decision models depend on S¹.
My read: The 9.5x figure is a maximum, not an average, and every performance and exactness claim in this paper is self-reported with no independent verification. The exactness guarantee, that every branch matches an isolated forward pass "in exact arithmetic," is a strong mathematical assertion, but the paper has not been peer-reviewed and no one outside the authors' labs has reproduced it S¹. The number I would watch is the total variation distance against the float32 reference, because if Flide's flattened execution introduces numerical drift, the whole premise of exact forking collapses.
The problem Flide addresses
Decision models such as Jev are part of a shift in how LLM agents work. Instead of generating text token by token, a decision model evaluates a tree of possible actions, scoring branches against each other before committing S¹. The preprint describes these as a promising approach for LLM agent systems S¹.
Standard inference engines were built for sequential generation, not tree-structured evaluation. When a decision model needs to score multiple branches that share a common prefix, engines like vLLM and SGLang either recompute that prefix for each branch, wasting GPU cycles, or batch the branches in ways that let attention leak across paths that should stay isolated, corrupting the probability scores S¹.
How Flide works
Flide's core idea is to flatten the logical tree of a decision request into a single token stream, then use a mechanism the authors call ForkAttention to manage which tokens attend to which S¹. Instead of running separate forward passes for each branch, Flide packs them into one pass and uses GPU-resident indices to enforce the dependency structure, the rule that branch A's tokens should never influence branch B's scores S¹.
ForkAttention operates across both standard attention layers and recurrent Gated DeltaNet layers, a newer architecture that maintains state across longer sequences without the full memory cost of traditional attention S¹. The engine also supports resuming from checkpoints saved in earlier passes, so an agent whose history grows over multiple turns does not need to reprocess everything from scratch S¹.
The authors claim that in exact arithmetic, every branch evaluated through this flattened approach produces identical results to running that branch alone in a separate forward pass S¹. In their evaluations, Flide stays closest to a float32 reference in total variation distance compared with the alternatives tested S¹. Total variation distance measures how much the output probability distribution drifts from the baseline.
What would have to hold
The preprint is categorised under cs.AI and cs.LG on arXiv and has not been peer-reviewed S¹. The 9.5x speedup is a best-case figure, not a guaranteed average across all workloads, and the paper does not report how performance scales as the tree widens or deepens. The exactness guarantee depends on "exact arithmetic," a theoretical condition that real GPUs, which use finite-precision floating point, do not fully meet. The code is public but barely watched: the OmniJev/Flide repository on GitHub, created 24 October, holds 2 stars and carries an Apache 2.0 licence, with topics tagging it for CUDA, Gemma, and KV-cache work P⁴. A separate repository, FrontiersMindAI/decisionformer, offers tooling to fine-tune and serve decision models on any Hugging Face model, with 39 stars and the same licence, created 22 October P³. Neither repository has been independently audited.
For a team building LLM agents that use decision models, the first thing to check is whether Flide's flattened execution preserves the probability rankings your application depends on. Run a single decision query through both Flide and a baseline forward pass, compare the top-k action scores, and look for any reordering. If the rankings diverge, the speedup is not worth the risk. The next checkpoint is whether the authors or a third party publish benchmarks on larger models and wider trees, which would test whether the 9.5x figure holds beyond the configurations in the preprint.
Sources: S1 — Flide: Accelerating Decision Inference with Exact State Forking · P2 — Flide: Accelerating Decision Inferencewith Exact State Forking · P3 — FrontiersMindAI/decisionformer · P4 — OmniJev/Flide · P5 — huskydoge/parcae-zoh-exact
Written from 5 sourced items, 4 of them primary.
