Serving Decision Models Fast: Flide's State Forking, Where It Wins, and Its Limits
When several questions share one context, Flide computes the shared state once and forks it per question with answers identical to asking alone; up to 9.5x faster than vLLM and SGLang at best, roughly 1.3x to 3.3x on average for shared-state workloads, and only about 1x to 3x with one question per input.