AGORA：一种基于档案的代理式工作场景文档推理基准
AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning
摘要
大型语言模型越来越多地被用作“智能体”，它们能够对文档进行推理处理，而非仅依据固定知识来回答问题。我们研究的是基于档案数据的推理过程：在大量杂乱的职场文件中找到有限的证据，解决不同领域之间术语、单位和时间规范上的不一致问题，从而得出答案。现有的评估标准只涵盖了这一场景的部分内容，没有哪个标准同时强调基于档案数据的推理、智能体的探索能力以及跨领域的覆盖能力。我们引入了Agora这个评估基准，它将362个问题与8个领域中的9,664份真实文档以及372M个标记词结合在一起，这些数量远远超过了任何模型的上下文窗口容量，因此智能体必须进行有意识的探索，而不是全面扫描。Agora是通过一种结合了跨文档任务整合、防止信息泄露的加密技术以及难度筛选的智能体处理流程构建而成的。对八种模型进行评估后发现，这些任务仍然难以完成：即使是最强大的模型也仅能达到59.4%的准确率，不同领域之间的表现差异也相当大。
English Abstract
Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing benchmarks address only parts of this setting and none jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage. We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window, so agents must explore deliberately rather than scan exhaustively. Agora is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering. Evaluating eight models, we find the task far from solved: even the strongest reaches only 59.4% accuracy, with notable variation across domains.