TokenPilot：适用于大语言模型智能体的高效缓存式上下文管理方案
TokenPilot: Cache-Efficient Context Management for LLM Agents
摘要
由于LLM代理在长时范围内进行推理，因此上下文的积累会显著增加推理成本。现有的方法通常采用文本剪枝或动态内存淘汰的方式来减少 tokens 的使用量；然而，这些方法对序列结构的无限制修改会导致布局的混乱，从而引发前缀不匹配和缓存失效的问题。这就形成了一个关键性的权衡：如何在文本稀疏性与提示缓存的连续性之间找到平衡。为了解决这一问题，我们提出了TokenPilot这一双粒度上下文管理框架。在全局层面，Ingestion-Aware Compaction机制能够稳定提示前缀，并消除输入过程中出现的环境噪声。而在局部层面，Lifecycle-Aware Eviction则能够监控上下文片段的剩余效用，只有当任务相关性消失时才进行内容片段的淘汰操作。在PinchBench和Claw-Eval测试平台上的实验表明，TokenPilot在孤立模式下可将成本降低61%，在连续模式下可降低61%和87%；同时，其性能仍然与现有系统相当。TokenPilot已集成到LightMem2中，相关代码可以在https://github.com/zjunlp/LightMem2下载。
English Abstract
As LLM agents are deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimize token footprints; however, their unconstrained sequence mutations alter layouts, introducing prefix mismatches and cache invalidation. This reveals a critical trade-off between text sparsity and prompt cache continuity. To address this, we present TokenPilot, a dual-granularity context management framework. Globally, Ingestion-Aware Compaction acts as a framework harness to stabilize prompt prefixes and eliminate open-world environmental noise at the ingestion gate. Locally, Lifecycle-Aware Eviction monitors the ongoing residual utility of context segments, enforcing a conservative batch-turn schedule to offload content segments only when task relevance expires. Experiments on PinchBench and Claw-Eval under both isolated and continuous modes demonstrate that TokenPilot reduces costs by 61% and 56% in isolated mode, and 61% and 87% in continuous mode, while maintaining competitive performance compared to prior systems. TokenPilot has been integrated into LightMem2 at https://github.com/zjunlp/LightMem2.