Qwen-RobotWorld技术报告：通过语言驱动的视频生成来实现具身世界的统一建模

Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

▲ 14 💬 1 2026-06-16

Jie Zhang, Xiaoyue Chen, Anzhe Chen, Chenxu Lv, Deqing Li, Gengze Zhou, Hang Yin, Haoqi Yuan, Haoyang Li, Jiahao Li, Jiazhao Zhang, Jingren Zhou, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Pei Lin, Qihang Peng, Shengming Yin, Tianhe Wu, Tianyi Yan, Xiao Xu, Yan Shu, Yanran Zhang, Ye Wang, Yi Wang, Yilei Chen, Yixian Xu, Yiyang Huang, Yuxiang Chen, Zekai Zhang, Zhendong Wang, Zhixing Lei, Zhixuan Liang, Zihao Liu, Zikai Zhou, Xiong-Hui Chen, Chenfei Wu

arXiv HF 原文 GitHub

摘要

我们介绍了Qwen-RobotWorld，这是一种基于自然语言条件的视频世界模型，旨在实现智能体的行为控制。通过将自然语言作为统一的操作接口，该模型能够基于当前的观测数据，预测出未来可能出现的视觉轨迹。这一模型可以应用于机器人操控、自主驾驶、室内导航以及人机交互等领域。这种统一的设计方式带来了三个有前景的应用方向：用于策略训练的合成数据生成、适用于策略评估的可扩展虚拟环境，以及由语言引导的机器人控制策略。这一成果是通过以下三部分设计实现的：a) 采用MLLM动作编码技术的双流MMDiT模型，其中一个60层的双流扩散变换器将固定的Qwen2.5-VL语义信息与视频VAE潜在表示相结合；b) 包含860万帧视频-文本数据的“智能体世界知识”库，其中包含超过20种智能体行为和500多种动作类别的映射信息；c) 一种两阶段训练策略，先学习通用的视觉先验知识，然后再在统一的语言接口下实现特定领域的专业知识。大量的实验结果表明，该模型的性能非常出色：在EWMBench和DreamGen Bench测试中排名首位，并且在WorldModelBench和PBench测试中的表现优于所有开源模型。此外，在RoboTwin-IF基准测试上的零样本分析也证明了该模型具有强大的泛化能力和多视图一致性。

English Abstract

We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.