以对象为中心的残差RL技术用于零样本真实场景VLA增强
Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement
摘要
视觉-语言-动作模型能够在各种操控任务中实现泛化能力，但其基于模仿学习的策略在精确的物理操作中仍存在稳定性问题，因为执行过程中会不断出现错误；那么，一种完全在模拟环境中训练的强化学习策略能否提升真实场景中的视觉-语言-动作模型的鲁棒性呢？残差强化学习方法可以在冻结的视觉-语言-动作模型基础上学习出有效的修正策略，从而提供一种自然的解决方案。不过，现有方法面临一个根本性的挑战：特权状态方法需要通过有损蒸馏来实现应用；基于图像的方案则存在视觉领域上的差异；而真实的强化学习则成本较高且不安全。我们提出了一种以物体为中心的残差强化学习框架，该框架利用物体的姿态信息来优化视觉-语言-动作行为，从而实现一个能够在模拟环境与现实环境之间稳定转换的观测空间。为了使两种环境能够相互衔接，我们还在模拟环境中重复使用相同的远程操作演示数据，以训练出与真实场景中的视觉-语言-动作模型相对应的模型。残差强化学习策略仅在模拟环境中训练，同时引入姿态噪声和dropout技术，然后就可以零样本地应用于真实机器人上。在针对真实Franka Research 3机器人的五个操控任务中，我们的方法将成功率从42%提升到了76%，而且这些改进后的行为模式还可以被再次用于重新训练基础视觉-语言-动作模型，无需额外的远程操作。项目页面：https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/
English Abstract
Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/