人类共有的感知能力
Human Universal Grasping
摘要
人类能够轻松抓取物体，而多指机器人则远远达不到这种通用性水平。我们认为，获取机器人抓取数据的自然来源就是人类——因为人类每天都会接触成千上万的物体。我们提出了HUG模型，该模型能够从立体相机拍摄的单个RGB-D图像中生成多种人类抓取的姿态。通过使用智能眼镜，我们首先收集了100万张人类抓取的图片数据，这些图片涵盖了100万帧画面，时间跨度达27.8小时，共包含6,707个物体实例，这些物体分布在41栋建筑中。接下来，为了模拟人类自然的抓取行为，我们的模型结合了RGB图像和深度信息，从而生成出由手腕移动、手腕旋转以及MANO手部姿态共同决定的抓取姿态。预测出的抓取姿态可以应用于各种机器人手中，从而实现在日常场景中无需预先训练即可进行抓取操作。为了标准化评估标准，我们创建了一个新的仿真基准测试集HUG-Bench，该测试集包含来自五个不同几何类别的90个未见过的物体，这些物体的大小各不相同，且所有物体都采用度量级3D网格格式表示。我们在真实环境中对HUG模型进行了测试，所使用的测试集包括30个物体，且测试环境包含了多种立体相机配置、不同形态的机器人结构以及家庭环境。结果表明，HUG模型在复杂的物体抓取任务中，其性能分别比现有最先进的抓取模型提升了23%和34%。我们的代码、数据、基准测试集以及交互式演示均可在我们的网站上找到：https://grasping.io/
English Abstract
Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality. We argue that the most natural source of robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated benchmark, HUG-Bench, of 90 unseen objects from five geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras, robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data, benchmark, checkpoints, and an interactive demo are released on our website: https://grasping.io/