将机器人视为一种智能体：面向视觉-语言-动作模型的以机器人为中心的点云映射
See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models
摘要
视觉-语言-动作模型能够根据视觉观测和语言指令来预测机器人的行为。这些行为是在机器人自身的3D坐标框架中定义的，但大多数视觉-语言-动作模型都是从相机视角来观察场景的，这就导致场景的观测位置和行为的定义位置之间存在不匹配。在固定视点的情况下，这种不匹配是可以接受的，因为策略可以记住单一的观测与行为的映射关系；但当使用大规模数据集时，由于不同相机设置下的行为数据被汇总在一起，策略就需要在不同的视点下也能实现这种映射关系的泛化。我们通过采用以机器人为中心的点图来解决这个问题——点图中的像素存储了场景中各点的3D坐标，这些坐标属于机器人的坐标系。点图能够提供机器人的3D几何信息，同时又能保持预训练的2D视觉-语言-动作模型所期望的密集H×W网格结构，因此可以在不造成太大架构变化的情况下被整合到现有的视觉-语言-动作模型中。在RoboCasa实验中，点图能够提升pi0.5和SmolVLA的性能，其表现优于那些基于相机视角或3D感知的基准模型。在实际机器人实验中，当相机被移动到训练过程中未出现的位置时，点图相对于仅使用RGB数据的策略的优势更加明显。
English Abstract
Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.