连接点：通过强化学习训练长生命周期智能体，实现跨领域泛化能力
Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning
摘要
本研究提出了一个用于训练大型语言模型（LLMs）以实现“Connect the Dots”功能的通用框架。这种能力是长期运行的人工智能智能体所必需的元能力：当基于LLM的智能体被部署到某个环境中时，它会完成一系列任务，同时不断探索环境、从自身经验中学习，并逐步更新对环境的理解，从而在后续任务中取得更好的表现。该框架的主要组成部分包括：(1) 用于端到端强化学习的算法设计与基础设施，其中长序列中混合了完成任务和更新上下文的环节；(2) 用于激励和激发LLMs所具有的元能力的任务与环境，以及用于评估过程中准确衡量进展的方法。我们展示了该框架的概念验证实现方式，包括一种具有细粒度奖励分配的GRPO风格强化学习算法，以及针对该元能力而设计的任务与环境设置——这些设置并非基于特定领域特性或传统的任务级强化学习机制。实验结果表明，端到端强化学习训练在“Connect the Dots”场景中非常有效，同时也证明了所获得的元能力具有跨域泛化能力——无论是在训练领域内、不同领域之间，还是从“Connect the Dots”场景到其他类似Ralph-loop场景中。我们对“Connect the Dots”的研究整合了多项先前研究成果，为推进LLMs和人工智能智能体的发展提供了新机会。为了促进进一步的研究与应用，我们在https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod上发布了我们的实现代码。
English Abstract
This work presents a general framework for training large language models (LLMs) to "Connect the Dots" (CoD), a meta-capability required by long-lifecycle agents: as an LLM-based AI agent gets deployed in an environment, it solves a long sequence of tasks while continuously exploring the environment, learning from its own experiences, and iteratively self-updating its context about the environment, thereby achieving progressively better performance on future tasks conditioned on the updated context. Major components of the CoD framework include: (1) algorithm design and infrastructure for end-to-end reinforcement learning (RL) with long rollout sequences interleaving solve-task and update-context episodes; (2) tasks and environments for incentivizing and eliciting the targeted meta-capability in LLMs during training, as well as for faithfully measuring progress during evaluation. We present proof-of-concept implementations of the CoD framework, including a GRPO-style RL algorithm with fine-grained credit assignment, as well as tasks and environments tailored to the targeted meta-capability (rather than domain-specific LLM capabilities or standard task-by-task RL). Empirical results validate the efficacy of end-to-end RL training in the CoD setting, and demonstrate the potential for out-of-distribution generalization -- within the training domains, across different domains, and from CoD to Ralph-loop settings -- of the elicited meta-capability. Our investigation of CoD connects several lines of prior works, and opens up new opportunities for advancing LLMs and AI agents. To facilitate further research and applications, we release our implementations at https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod.