> ## Content Index
> Fetch the complete content index at: https://www.notatechguy.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# SGLang cache system doubles AI agent turns on NVIDIA chips
- URL: https://www.notatechguy.com/sglang-cache-system-doubles-ai-agent-turns-on-nvidia-chips/
- Published: 2026-11-21T15:50:29.000Z
- Updated: 2026-11-21T15:50:29.000Z
- Description: TideCache, built on SGLang, cuts per-token latency up to 59% on NVIDIA GB300 and GH200 by streaming KV cache layers between GPU and host memory.
- Author: Marcello Babbili
- Tags: Technology & AI, Nvidia, AI Models, AI Chips

Jinbin Luo and colleagues posted a preprint on arXiv describing a serving system that completes up to 2.04 times as many agent turns as SGLang's existing cache setup on the same hardware [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). The system, called TideCache, targets a memory bottleneck that has grown as open-weight models increasingly mix two types of attention layers, each with different demands on GPU memory [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). Nobody outside the authors' lab has checked these numbers yet. The paper is a preprint, not peer-reviewed [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com), and the evaluation runs only on NVIDIA GB300 and GH200 superchips across seven model-and-platform configurations [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com).

**My read:** The 2.04x agent-turn figure is the number that matters, because agent workloads are where KV cache pressure bites hardest: long conversations, repeated context, multi-turn reasoning. But I would hold judgement on the 20 to 59 percent latency claim until I saw results on hardware beyond NVIDIA's superchip line. The baseline, SGLang with HiCache, is a reasonable comparison, but HiCache is itself an evolving feature, so the gap could narrow if SGLang's own development catches up.

### The memory squeeze hybrid models create

Modern open-weight language models increasingly use hybrid architectures, mixing global-attention layers (which store every token's key-value pairs) with bounded-state layers (which keep only a fixed window of recent tokens) [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). The global-attention layers eat GPU memory. The bounded-state layers do not. That imbalance makes it hard to pack enough requests onto a single chip.

The KV cache — the stored key-value pairs that spare a model from reprocessing earlier tokens, is the biggest consumer of HBM on the GPU during inference. When the cache outgrows available HBM, serving systems either evict entries or spill them to slower host memory, both of which add latency. The open-source serving ecosystem has been racing to solve exactly this kind of memory pressure.

### Three components, one pool

TideCache has three parts. The Transparent Cache Pool moves physical memory pages between different cache components without changing virtual addresses, meaning existing backend code works as-is [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). The Cache Streamer takes advantage of the time bounded-state layers spend computing to pull the next global-attention layer's prefix into GPU slots that are shared across those layers, so data transfers happen while computation is underway [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). The Request Scheduler decides for each request whether to keep everything in GPU memory (resident mode) or fetch layers on demand (streamed mode), choosing based on request size and how deep the queue is [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com).

The combined effect: HBM freed by streaming can hold other cache components or additional concurrent requests [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com).

### What the numbers say, and what they do not

Against SGLang with HiCache on the same hardware, TideCache reduces overall P50 per-token latency by 20 to 59 percent and improves agent P50 time-to-first-token by 1.40 to 3.05 times [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). The system was implemented in SGLang and evaluated on NVIDIA GB300 and GH200 across seven model-and-platform configurations [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). Those are author-reported figures from a preprint that has not been peer-reviewed [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com).

The paper does not claim results on hardware other than GB300 and GH200, and the performance gains are measured relative to a specific baseline rather than against every available caching strategy [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). A separate effort, LMCache, which describes itself as a fast KV cache layer for LLMs, has accumulated over 12,000 stars on GitHub since May 2024 [P⁵](https://github.com/lmcache/LMCache?ref=notatechguy.com), showing how active and competitive this space has become.

For a team running agent workloads on NVIDIA superchips through SGLang today, the thing to check is whether your workload's ratio of global-attention to bounded-state layers matches one of the seven configurations the authors tested. If it does, the prefetch-and-stream approach could raise throughput. If your model architecture differs, the gains are untested.

The authors have not stated when or whether TideCache will be merged into the upstream SGLang repository [S¹](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com). The preprint is available on arXiv now, posted 19 November 2026.

---

*Sources: [S1 — TideCache: Efficient Hybrid LLM Serving with Transparent Cache Managem](https://arxiv.org/abs/2611.26619v1?ref=notatechguy.com) · [P2 — 1Introduction](https://arxiv.org/abs/2611.26619?ref=notatechguy.com) · [P3 — UIUC-MLSys/HyGen](https://github.com/UIUC-MLSys/HyGen?ref=notatechguy.com) · [P4 — 1Introduction](https://arxiv.org/html/2611.26619?ref=notatechguy.com) · [P5 — LMCache/LMCache](https://github.com/lmcache/LMCache?ref=notatechguy.com)*

---

*Written from 5 sourced items, 4 of them primary.*

![TideCache performance vs SGLang with HiCache best case](https://storage.ghost.io/c/6e/89/6e896869-22ef-4281-a213-b4c462c17cff/content/images/2026/11/chart_e539e488824f51b96b05.png)

## More from Not A Tech Guy

- [OpenAI's 'Silent Consensus' asks who keeps control as AI grows](https://www.notatechguy.com/openai-s-silent-consensus-asks-who-keeps-control-as-ai-grows/)
- [Hugging Face harness adapters lift Qwen 27B score 39% to 55%](https://www.notatechguy.com/hugging-face-harness-adapters-lift-qwen-27b-score-39-to-55/)
- [Google's Duplex now calls businesses for you inside Gemini](https://www.notatechguy.com/google-s-duplex-now-calls-businesses-for-you-inside-gemini/)