CoffeeBench：在异质多智能体经济环境中对长期视角的LLM智能体进行基准测试
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
摘要
随着LLM智能体能够完成更长的时间跨度内的任务，评估它们在经济系统中的表现变得愈发重要。与那些主要评估单个智能体与被动环境互动的现有基准不同，经济系统本质上是由多个智能体构成的，这些智能体需要相互沟通、协商并进行交易，同时各自追求自己的目标，且这个过程需要持续很长时间。我们提出了CoffeeBench这一基准测试工具，用于评估由不同企业组成的长期多智能体经济中的LLM智能体。在CoffeeBench中，两名农民、两名咖啡烘焙商和两名零售商在90天的模拟过程中自主运营他们的业务，他们通过沟通和交易来最大化总净收入，同时还需管理资金、库存和价格策略。被评估的模型中有一个智能体负责控制咖啡烘焙业务，而其他企业则由固定的参考智能体来控制。在多个最近的公开和私有LLM模型中，所有模型都优于不采取任何行动的被动基准模型，大多数模型都能实现正净收入。对智能体行为的分析表明，它们在长期经济互动中存在显著差异：表现更好的模型会更积极地与其他企业沟通，而Claude~Haiku~4.5则表现出“闲置状态”的失败模式，尽管能够做出合理的评估和计划，却始终选择不采取行动。我们公开了我们的代码和智能体行为轨迹，以支持未来的研究。
English Abstract
As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. In CoffeeBench, two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation, each seeking to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. The evaluated model controls one coffee roaster, while the remaining firms are controlled by fixed reference agents. Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions, with most achieving positive net income. Analysis of agent behavior reveals substantial differences in long-horizon economic interaction: higher-performing models communicate more actively with other firms, whereas Claude~Haiku~4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. We release our code and agent trajectories to support future research.