Track2View：通过配对的三维点轨迹技术，实现与4D一致性的摄像机控制视频生成
Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks
摘要
重新渲染来自不同相机视角的视频时，需要确保输出结果能够遵循原始相机的运动轨迹，同时保留每一帧中场景的外观和动态效果。现有的方法通常依赖于每帧的姿态信息、噪声点云渲染技术，或者隐式学习到的对应关系等方法来实现这一目标。但这些方法都无法在源像素和目标像素之间建立一种明确且时间上连续的联系。我们提出的Track2View方法则基于这样的原理：将视频渲染过程与3D点轨迹相结合。这些点轨迹代表了场景中各个点的稀疏运动路径，这些路径可以被视为源视角和目标视角之间的对应关系。这种对应关系使得内容在时间和空间上的呈现具有连续性，从而能够确定哪些内容应该出现在哪里以及何时出现。Track2View的核心思想在于使用一种双视图点轨迹处理技术，通过无参数的几何运算和学习的时序聚合机制，将视觉上下文从源视角传输到目标视角，从而无需记忆具体的运动规律即可适用于任何相机视角。我们还引入了一种数据预处理流程，该流程通过在一个时间上连续的多相机视图对上运行3D点跟踪算法，来提取一一对应的点轨迹。在一项包含400个视频样本的测试案例中，Track2View在视觉质量、视角同步性和相机精度方面均取得了领先成果：旋转误差减少了30-65%，平移误差则减少了61-72%。更多相关信息可以访问以下链接：https://qjizhi.github.io/track2view
English Abstract
Re-rendering an existing video from a novel camera viewpoint requires the output to follow the prescribed camera trajectory while preserving the appearance and dynamics of the original scene across every frame. Existing methods rely on per-frame pose embeddings, noisy point-cloud renderings, or implicit learned correspondences, none of which provides an explicit, temporally continuous link between source and target pixels. We propose Track2View, which conditions a video diffusion transformer on paired 3D point tracks: sparse trajectories of scene points projected into both the source and target camera views. These tracks provide explicit spatiotemporal correspondences that are temporally continuous by construction, encoding what content should appear where and when. At the core of Track2View is a dual-view track conditioner that transfers visual context from source to target view through parameter-free geometric operations and learned temporal aggregation, ensuring generalization to arbitrary camera trajectories without memorizing specific motions. We further introduce a data curation pipeline that extracts one-to-one track correspondences by running a 3D point tracker on temporally concatenated multi-camera view pairs. On a 400-video benchmark spanning static and dynamic scenes, Track2View achieves state-of-the-art results across visual quality, view synchronization, and camera accuracy, reducing rotation error by 30-65% and translation error by 61-72% relative to leading baselines. Project page is available at this https URL: https://qjizhi.github.io/track2view