番石榴:一种有效且通用的用于实体操作的工具
Guava: An Effective and Universal Harness for Embodied Manipulation
摘要
在大规模视觉-语言数据上训练的语言模型具有强大的潜力,可以用于实现具身智能体。通过利用这些模型来构建具身工具,就可以以一种有前景的方式替代端到端的视觉-语言-动作系统,因为这种方式能够将高级推理能力与用于感知、规划和控制的外部模块相结合。不过,究竟什么因素使得这种集成方式能够有效实现具身操作,以及这种集成方式能在多大程度上提升各种推理模型的具身能力,目前仍不清楚。在这项研究中,我们提出了Guava这一用于构建具身工具的框架,该框架是通过对智能体工作流、动作空间和观察空间的设计空间进行系统研究而开发的。我们的研究指出,有效的具身智能体需要三个关键要素:迭代性的感知-推理-动作循环、语义化的动作抽象机制,以及多模态观测能力。为了了解这些设计原则是否适用于小型模型,我们开发了一个端到端的训练流程,该流程使用不到2K条在模拟环境中收集到的轨迹,将具身操作能力转化为4B开源模型。在模拟环境和真实环境中的实验结果表明,该模型的性能与先进的专有模型相当,同时还能很好地处理那些未见过的物体、新的指令以及长期任务。结果表明,设计良好的框架可以作为一种可扩展的、与模型无关的接口,帮助小型开源模型在较少的训练数据下实现强大的具身能力。
English Abstract
Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents. Harnessing models through embodied tools use offers a promising alternative to end-to-end vision-language-action systems by combining high-level reasoning with external modules for perception, planning, and control. However, it remains unclear what makes an effective harness for embodied manipulation, and to what extent such a harness can unlock embodied capabilities in a wide range of reasoning models. In this work, we present Guava, a harness framework for embodied tool use developed through systematic exploration of the design space of agent workflows, action spaces, and observation spaces. Our study identifies three key ingredients for effective embodied agents: iterative perception-reasoning-action loops, semantic action abstractions, and multimodal observations. To understand whether these design principles are universal even to small models, we develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model using fewer than 2K trajectories collected entirely in simulation. Experimental results in both simulation and real-world environments show performance comparable to frontier proprietary models while exhibiting strong generalization to unseen objects, novel instructions, and long-horizon tasks. Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation, enabling strong emergent embodied capabilities in compact open-source models with minimal training data.