利用全局地图和局部视角进行多视图3D推理的丰厚回报
Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views
摘要
多视图3D视觉问答系统（MV3D-VQA）需要将各个部分的信息整合成连贯的3D场景表示，同时选择合适的视角来进行多步骤的空间推理。然而，目前的多模态大语言模型通常是通过稀疏的、基于答案层面的监督来训练的，这往往会导致跨视图的推理结果不一致，且视角选择也不够稳定。我们提出了DR-MV3D这一基于地图的监督学习框架，该框架提供密集且可验证的奖励信息，以引导推理过程。我们的方法将MV3D-VQA分解为三个步骤：(i) 以全局地图为基准的视角构建；(ii) 根据问题需求来规划视角变化路径；(iii) 以个体视角为基准进行答案预测。为了让中间步骤能够通过自动学习来实现，我们引入了两种奖励机制：一种是全球一致性奖励，用于确保预测的地图与冻结的3D视觉基础模型所生成的几何一致的目标相匹配；另一种则是局部轨迹奖励，用于指导视角的选择。我们通过轨迹级别的策略优化方法来优化整个流程。在MindCube、VSI-Bench和BLINK（MV）上的实验表明，DR-MV3D显著优于其他多图像基线模型，从而证明了基于过程的密集监督对于多视图3D推理的有效性。
English Abstract
Multi-view 3D Visual Question Answering (MV3D-VQA) requires integrating partial observations into a coherent 3D scene representation and selecting informative viewpoints for multi-step spatial reasoning. However, current multimodal LLMs are typically trained with sparse, answer-level supervision, which often yields inconsistent cross-view reasoning and brittle view selection. We present DR-MV3D (Dense Reward for MV3D-VQA), a map-grounded learning framework that provides dense, verifiable rewards to supervise the reasoning process. Our approach decomposes MV3D-VQA into (i) allocentric global map construction, (ii) question-conditioned view-trajectory planning, and (iii) egocentric grounding for answer prediction. To make intermediate steps learnable without manual annotations, we introduce two rewards: a global consistency reward that aligns the predicted map with geometry-consistent pseudo targets from frozen 3D vision foundation models (e.g., VGGT + SAM3), and a local trajectory reward that supervises ordered viewpoint selection. We optimize the full pipeline with trajectory-level policy optimization (GRPO). Experiments on MindCube, VSI-Bench, and BLINK (MV) show that DR-MV3D consistently improves over strong multi-image baselines, supporting the effectiveness of process-level dense supervision for multi-view 3D reasoning.