唐格拉姆：解锁非均匀KV缓存压缩技术，以实现高效的多轮次LLM服务
Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
摘要
多轮对话处理系统中，对话历史记录被存储在键值缓存中。随着每一轮的推进以及每个用户的参与，键值缓存的大小会不断增大，甚至超过模型本身的权重。此时，内存成为了限制吞吐量的关键因素，而非计算能力。非均匀键值压缩方式能够更精确地分配资源，因为它允许不同注意力头之间分配不同的资源预算。不过，这种方案仍然不切实际：现代的服务框架通常假设所有注意力头的键值长度都是相同的。因此，这种不均匀性会导致内存碎片问题，需要花费高达25%的时间来重新整理分散的内存块。此外，这种不均匀性还会使得GPU的工作负载分布出现偏差，从而导致解码延迟增加1.7倍，或者每次解码过程中消耗15-20%的计算资源。我们注意到，这种不均匀性其实可以在运行时被发现：每个注意力头的存储方式遵循一种两级结构规律——即输入不变的情况下，每个注意力头的资源分配比例是有上限的。这种规律可以通过少量样本数据进行离线校准。基于这一发现，我们提出了Tangram框架。该框架能够静态地解决传统系统需要动态处理的难题：预算预留功能可以在调度时确定每个注意力头在压缩后的资源占用情况，从而消除内存碎片问题；类似预算的注意力头可以被合并到独立的页面表中，从而实现可回收的内存；而提前的负载均衡功能则可以在零时间规划下实现平衡的GPU分区分配。Tangram框架可以在vLLM上实现，它可以作为现有非均匀压缩方法的替代方案，同时其准确性与现有方法相当，但端到端吞吐量则可以提高2.6倍。我们的实现代码可以在https://github.com/aiha-lab/TANGRAM下载。
English Abstract
Multi-turn LLM serving accumulates dialogue history whose Key-Value (KV) cache grows with every turn and every user, quickly exceeding the model weights themselves and making memory -- not compute -- the binding constraint on throughput. Non-uniform KV compression, which allocates heterogeneous budgets across attention heads, preserves accuracy far better than uniform schemes, yet remains impractical: modern serving stacks assume identical KV lengths across heads, so heterogeneity traps freed memory as page fragmentation, spends up to 25% of prefill time reclaiming scattered pages, and skews GPU workloads that inflate decode latency by up to 1.7times or burn 15--20% of each decode step on re-planning. We observe that this heterogeneity need not be discovered at runtime: head-wise retention follows a two-level structural regularity -- an input-invariant head ranking with narrowly bounded per-head ratios -- that can be calibrated offline from as few as 50 samples. Building on this insight, we present Tangram, a serving framework that statically resolves what prior systems handle dynamically: Budget Reservation fixes each head's post-compression footprint at scheduling time, eliminating page reclamation; Ragged Paging clusters similar-budget heads into independent page tables, turning fragmentation into reclaimable memory; and Ahead-of-Time Load Balancing precomputes balanced GPU partitions with zero runtime planning. Implemented on vLLM, Tangram serves as a drop-in substrate for existing non-uniform compression methods, matching their accuracy while improving end-to-end throughput by up to 2.6times over the full-KV baseline. Our implementation is publicly available at https://github.com/aiha-lab/TANGRAM.