‹ 返回 2026-06-19

作为全模态理解基础的本土主动感知机制

Native Active Perception as Reasoning for Omni-Modal Understanding

▲ 10 💬 1 2026-06-19

Zhenghao Xing, Ruiyang Xu, Yuxuan Wang, Jinzheng He, Ziyang Ma, Qize Yang, Yunfei Chu, Jin Xu, Junyang Lin, Chi-Wing Fu, Pheng-Ann Heng

摘要

用于长视频理解的被动模型通常采用“全面观看”模式,无论查询难度如何都统一处理每一帧图像,导致计算成本随着视频时长增加而上升。虽然已有交互式框架出现,但这些框架往往依赖全局预扫描机制,其上下文处理成本仍然与视频长度成正比。我们提出了OmniAgent——首个原生多模态智能体,它将视频理解过程视为基于POMDP的迭代式“观察-思考-行动”循环。OmniAgent能够按需执行各种操作,从而将音频和视觉信息转化为可持久保存的文本信息,有效将推理复杂性与视频总时长分离开来。为了实现这一目标,我们引入了两种技术:一是基于多轨迹合成的代理式监督微调方法,通过双阶段质量控制机制实现高效优化;二是基于TAURA的代理式强化学习机制,利用turn级别熵值来引导注意力集中在关键发现环节上。重要的是,OmniAgent在测试过程中具有良好的扩展性,即随着推理次数的增加,性能也会提升,这证明了主动感知策略的有效性。在十个基准测试中的实验结果表明,OmniAgent在开源模型中达到了最先进的性能水平。特别是在LVBench测试中,我们的7B参数智能体表现优于规模大10倍的Qwen2.5-VL-72B模型(50.5%对47.3%)。

English Abstract

Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10times larger Qwen2.5-VL-72B (50.5% vs. 47.3%).