GraniKV: Asymmetric KV-Cache Paging for Shared-Prefix Agents

This episode examines GraniKV, a KV-cache paging scheme for multi-agent workloads where many requests share a very long prompt prefix. It argues that engines like SGLang treat the shared prefix and each request's private suffix the same way, using 16-token pages, even though the two want opposite things from memory. The discussion first covers the background: PagedAttention, RadixAttention prefix reuse, and the point that reuse saves prefill compute but not the repeated memory reads during decode. It then separates two prior kernel ideas, cascade attention, which reads the prefix once to cut memory traffic, and Hydragen, which batches queries into dense matrix multiplies but needs a contiguous prefix. GraniKV's claim is about storage and integration rather than new kernels. It keeps each prefix as one contiguous slab in a reserved "hot" pool, keeps suffixes at single-token granularity in a "cold" pool, and chooses an attention backend at each decode step. The episode tests where the reported 2.16x throughput gain over stock SGLang at a 16K-token prefix comes from, asking whether the layout or the kernel does the work, with a self-built cascade baseline as the comparison. It is useful for listeners who want to understand why prefix-heavy agent serving is memory-bound and how storage layout interacts with attention kernels.

Sources:

1. GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix — Jinhyun Jeon, Sungjoo Yoo, 2026

http://arxiv.org/abs/2608.15584v1

2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2024

https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs

4. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar, 2024

https://scholar.google.com/scholar?q=vAttention%3A+Dynamic+Memory+Management+for+Serving+LLMs+without+PagedAttention

5