集群、路径、升级：基于成本意识的LLM服务级级架构
Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
摘要
在生产环境中有效部署大型语言模型，必然需要在准确性与成本之间做出权衡。通常，开发人员会选择一种单一模型来处理查询，但这种模型要么对于简单查询来说成本过高，要么对于复杂查询来说则不够可靠。为了应对这一挑战，我们提出了一种两阶段级联解决方案。第一阶段会对传入的查询进行分组，并将每个群组分配到最经济高效的模型中。这一路由过程的成本预算可以通过离线调整的可解释超参数来设定。第二阶段则引入了质量评估机制：当第一阶段生成的输出被判定为质量较低时，该查询就会被转交给更强大的模型来处理。这样就能确保只有那些复杂或可靠性较低的查询才会被送到昂贵的模型中。在测试数据集上，这种级联系统能够保留最强模型的97-99%的准确率，同时降低每个输出令牌所花费的时间。该系统仅需任务相关的标签即可工作，而且无需手动重新配置，就能适应模型池的变化。
English Abstract
Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.