Jinbin Luo and colleagues posted a preprint on arXiv describing a serving system that completes up to 2.04 times as many agent turns as SGLang's existing cache setup on the same hardware S¹. The system, called TideCache, targets a memory bottleneck that has grown as open-weight models increasingly mix two types of attention layers, each with different demands on GPU memory S¹. Nobody outside the authors' lab has checked these numbers yet. The paper is a preprint, not peer-reviewed S¹, and the evaluation runs only on NVIDIA GB300 and GH200 superchips across seven model-and-platform configurations S¹.

My read: The 2.04x agent-turn figure is the number that matters, because agent workloads are where KV cache pressure bites hardest: long conversations, repeated context, multi-turn reasoning. But I would hold judgement on the 20 to 59 percent latency claim until I saw results on hardware beyond NVIDIA's superchip line. The baseline, SGLang with HiCache, is a reasonable comparison, but HiCache is itself an evolving feature, so the gap could narrow if SGLang's own development catches up.

The memory squeeze hybrid models create

Modern open-weight language models increasingly use hybrid architectures, mixing global-attention layers (which store every token's key-value pairs) with bounded-state layers (which keep only a fixed window of recent tokens) S¹. The global-attention layers eat GPU memory. The bounded-state layers do not. That imbalance makes it hard to pack enough requests onto a single chip.

The KV cache — the stored key-value pairs that spare a model from reprocessing earlier tokens, is the biggest consumer of HBM on the GPU during inference. When the cache outgrows available HBM, serving systems either evict entries or spill them to slower host memory, both of which add latency. The open-source serving ecosystem has been racing to solve exactly this kind of memory pressure.

Three components, one pool

TideCache has three parts. The Transparent Cache Pool moves physical memory pages between different cache components without changing virtual addresses, meaning existing backend code works as-is S¹. The Cache Streamer takes advantage of the time bounded-state layers spend computing to pull the next global-attention layer's prefix into GPU slots that are shared across those layers, so data transfers happen while computation is underway S¹. The Request Scheduler decides for each request whether to keep everything in GPU memory (resident mode) or fetch layers on demand (streamed mode), choosing based on request size and how deep the queue is S¹.

The combined effect: HBM freed by streaming can hold other cache components or additional concurrent requests S¹.

What the numbers say, and what they do not

Against SGLang with HiCache on the same hardware, TideCache reduces overall P50 per-token latency by 20 to 59 percent and improves agent P50 time-to-first-token by 1.40 to 3.05 times S¹. The system was implemented in SGLang and evaluated on NVIDIA GB300 and GH200 across seven model-and-platform configurations S¹. Those are author-reported figures from a preprint that has not been peer-reviewed S¹.

The paper does not claim results on hardware other than GB300 and GH200, and the performance gains are measured relative to a specific baseline rather than against every available caching strategy S¹. A separate effort, LMCache, which describes itself as a fast KV cache layer for LLMs, has accumulated over 12,000 stars on GitHub since May 2024 P⁵, showing how active and competitive this space has become.

For a team running agent workloads on NVIDIA superchips through SGLang today, the thing to check is whether your workload's ratio of global-attention to bounded-state layers matches one of the seven configurations the authors tested. If it does, the prefetch-and-stream approach could raise throughput. If your model architecture differs, the gains are untested.

The authors have not stated when or whether TideCache will be merged into the upstream SGLang repository S¹. The preprint is available on arXiv now, posted 19 November 2026.


Sources: S1 — TideCache: Efficient Hybrid LLM Serving with Transparent Cache Managem · P2 — 1Introduction · P3 — UIUC-MLSys/HyGen · P4 — 1Introduction · P5 — LMCache/LMCache


Written from 5 sourced items, 4 of them primary.

TideCache performance vs SGLang with HiCache best case

More from Not A Tech Guy