针对代理式和多模态LLM的上下文感知强化学习
Context-Aware RL for Agentic and Multimodal LLMs
摘要
大型语言模型在需要从长或复杂的上下文中找到那些微小但关键的线索时往往表现不佳，比如工具记录中的单行文字或图像中的细微细节。我们提出了ContextRL这一基于上下文感知的强化学习方法，通过一种间接的辅助目标来提升长距离推理能力和多模态处理能力。与仅关注最终答案不同，ContextRL让模型面对一个查询、一个答案以及两个高度相似的上下文，并奖励模型选择能够支持该查询-答案对的上下文的行为，从而鼓励更细致的上下文选择。我们在两个领域构建了对比性上下文数据：对于编码代理来说，轨迹作为上下文，通过条件过滤生成了1k对数据；对于多模态推理来说，图像作为上下文，通过生成式编辑和相似性搜索生成了7K对数据。在5个长距离基准测试中，ContextRL比标准GRPO平均提升了2.2%，在12个不同的视觉问答基准测试中则提升了1.8%。为了区分所提出的目标与额外数据的影响，我们与那些使用相同对比性上下文作为标准查询-上下文-答案示例的数据增强基线进行了比较。这些基线几乎没有带来任何改进，说明所获得的性能提升主要来自所提出的上下文选择目标，而非单纯的对比性数据。
English Abstract
Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an indirect auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query--answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone.