用于机器人控制的上下文环境建模
In-Context World Modeling for Robotic Control
摘要
现代视觉-语言-动作模型往往无法适应新的环境条件，比如不同的相机视角或机器人的形态变化。因为这些模型通常只基于当前的观测数据和语言指令来构建模型。它们忽略了系统配置这一变量，从而默认认为训练过程中所遇到的执行环境是固定的，因此需要在新环境中进行大量数据驱动的微调。在本研究中，我们提出了“上下文中的世界建模”框架，该框架将系统识别视为一种上下文适应问题。这种框架使得机器人策略能够从短暂的自我生成互动数据中自动推断出系统的关键参数，而无需特定于某个任务。与传统的上下文学习方法不同，后者依赖示范信息来指定要执行的任务，而“上下文中的世界建模”则利用上下文信息来理解系统的运作方式。通过在任务执行之前处理这些互动数据，模型能够隐式地掌握当前系统的动态特性，从而无需参数更新就能适应新的环境条件。在仿真环境和真实机器人平台上的大量实验表明，“上下文中的世界建模”在应对新的相机视角时，其性能明显优于传统VLA模型。
English Abstract
Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.