无政府状态下的分散推理代价
The Price of Anarchy in Disaggregated Inference
摘要
这种分散式的推理架构将“预填充”和“解码”两个阶段分离到不同的GPU池中，从而形成了多个相互竞争的“代理”，而这些代理共享相同的硬件资源。据我们所知，这是首次对这种架构进行正式的游戏论分析，我们以NVIDIA Dynamo为例进行了实证研究。我们将这种分散式服务模型分解为三个相互关联的博弈：一个是预填充池与解码池之间的双玩家资源博弈；另一个是针对分层KV缓存的自私缓存策略博弈；还有一个是涉及请求路由的、具有正外部性的拥堵博弈。我们实证验证了后两个博弈的情况；而预填充/解码资源博弈则通过分析方法进行了处理（详见第9.2节）。我们还研究了GPU饱和度如何引发博弈的收益结构变化：在饱和度较低时，自私行为导致的无序成本是有限的；而在饱和度较高时，超线性的延迟成本和缓存外部性则会导致我们的实证估计值PoA-hat上升（该值的定义见第6.4节）。基于这一分析，我们设计了一个自适应控制器，能够实时检测饱和度变化，并相应地调整路由参数，从利用缓存资源转向平衡负载的拥堵缓解策略。我们在配备了两款模型的3节点NVIDIA B200集群上实现了该框架——分别是Nemotron-4-340B（TP=8，全节点工作节点支持跨InfiniBand的KV传输）和Llama-3.1-70B（TP=4）。我们发现，在两种模型中都呈现了相同的三阶段PoA-hat结构，且第一个网格点处的参数值相同（C=128）。自适应路由机制使得每种模型都能达到更好的运行状态。我们最显著的成果体现在70B单节点/5线程的拓扑结构上：在饱和度较高的状态下，PoA-hat的值下降了3.1倍（从66.4降至21.5），而吞吐量成本仅增加了13%。在70B单节点/2线程的拓扑结构上，PoA-hat的值下降了2.2倍，而TTFT的P99值则下降了7.6倍（详见第8.5节）。
English Abstract
Disaggregated inference architectures physically separate prefill and decode phases onto distinct GPU pools, creating competing "agents" that share a fixed hardware budget. We provide, to our knowledge, the first formal game-theoretic analysis of this architecture, using NVIDIA Dynamo as a concrete case study. We model disaggregated serving as three coupled games: a two-player resource game between prefill and decode pools, a selfish caching game over the hierarchical KV cache, and a congestion game with positive externalities for request routing. We empirically validate the latter two; the P/D resource game is treated analytically (Section 9.2). We characterize how GPU saturation induces regime transitions that shift the game's payoff structure: below saturation, selfish behavior has bounded Price of Anarchy (PoA); at saturation, superlinear latency and cache externalities drive our empirical estimator PoA-hat (defined in Section 6.4) upward. Based on this analysis, we design an adaptive controller that detects saturation transitions in real time and adjusts routing parameters accordingly, shifting from cache-affinity exploitation to load-balanced congestion avoidance. We instantiate our framework on a 3-node NVIDIA B200 cluster running Dynamo with two models, Nemotron-4-340B (TP=8, full-node workers with cross-InfiniBand KV transfers) and Llama-3.1-70B (TP=4), and find the same three-regime PoA-hat structure with the same first post-knee grid point (C=128) on both models. Adaptive routing shifts each model to a better operating point. Our strongest result is on the 70B 1P/5D topology, where PoA-hat drops 3.1x (66.4 to 21.5) in the saturated phase at a 13% throughput cost. On the 70B 1P/2D, PoA-hat drops 2.2x and TTFT P99 drops 7.6x (see Section 8.5).