每日报告：用于评估搜索代理在日常搜索任务中的表现的一种开放式基准标准
DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks
摘要
搜索代理通常利用大型语言模型来协助完成复杂的信息检索任务，它们能够自主探索网络资源，并将相关信息整合成完整的回答。在评估这些搜索代理时，现有的评估标准主要针对那些在现实场景中使用概率较低的特定任务。此外，由于它们依赖较为粗略的任务级评估标准，因此评估结果往往难以被准确解读。为了弥补这一不足，我们引入了DailyReport这个开放式的评估基准，用于评估搜索代理在日常搜索任务中的能力。该基准包含150个开放式任务以及3,546个相关的评估标准，能够反映现实用户所关心的各种信息需求。每个任务被分解为多个子任务，并通过不同维度上的评估标准进行评判。通过这种分层式的性能评估方式以及以用户为中心的汇总方法，我们可以获得每个维度的可解释性较高的评分，同时还能得到用户的偏好评分。我们对17种搜索代理系统的评估结果表明，当前系统仍然未能满足用户的期望。为了促进未来的研究，我们的数据集和代码已公开在https://github.com/AGI-Eval-Official/DailyReport上。
English Abstract
Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses. For SAs evaluation, prior benchmarks mainly focus on specialized tasks that are unlikely to arise in real-world user scenarios. Moreover, their reliance on coarse task-level rubrics often limits evaluation interpretability. To bridge this gap, we introduce DailyReport, an open-ended benchmark to evaluate SA capabilities on daily search tasks. It contains 150 open-ended tasks with 3,546 associated rubrics, capturing widely discussed and timely information demands of real-world users. Each task is decomposed into subtasks and evaluated with cascade rubrics across disentangled dimensions. Through cascade performance attribution and user-centric aggregation, we derive highly interpretable scores for each dimension, along with a user preference score. Our results on 17 agentic systems show that current systems still fall short of users' expectations. To facilitate future research, our dataset and code are made publicly available at https://github.com/AGI-Eval-Official/DailyReport.