跟随轨迹：基于点跟踪的视频合成与运动控制
Go-with-the-Track: Video Compositing and Motion Control with Point Tracking
摘要
电影制作需要精确的运动控制与参考图像合成功能——这些功能通常被不同的方法分别处理。基于点-轨迹条件的图像到视频模型只能将内容插入到第一帧中，而基于参考图像的视频模型则缺乏对参考内容在不同帧中如何融合的精细空间-时间控制能力。 我们提出了Go-with-the-Track模型，该模型通过结合多个参考图像以及以参考图像为基准的点-轨迹信息来实现这两种功能。这种模型能够将生成的视频帧与参考图像进行精确匹配，从而实现整个视频中的精准合成与运动控制。 为了实现这一点，我们引入了具有空间感知能力的点-轨迹嵌入方式：这种嵌入方式使用逐坐标的MLP网络来编码点-轨迹的坐标信息，然后再进行时间上的池化操作。这种表示方式能够捕捉到每个点-轨迹的空间特征（作为唯一的标识符），而嵌入方式的相似性则与空间距离直接相关，从而提升了模型区分和关联点-轨迹的能力。我们将这些点-轨迹嵌入信息通过轻量级适配器引入到视频扩散Transformer模型中，从而解决了像素与块之间的分辨率不匹配问题，同时避免了简单点-轨迹子采样所带来的大量运动细节损失。 我们采用混合训练策略，在动态、静态以及合成场景视频数据集上进行联合训练，以提高运动控制的精度。实验表明，Go-with-the-Track模型能够在单一模型中实现出色的运动与参考图像控制效果，同时还能实现新的功能：基于多个参考图像的视频生成，以及针对静态和动态场景的相机控制功能。项目页面：https://eyeline-labs.github.io/Go-with-the-Track/
English Abstract
Filmmaking demands precise motion control and reference image compositing -- capabilities that existing methods treat separately. Point-track-conditioned image-to-video models restrict content insertion to the first frame, while reference-to-video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks -- extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the video. To achieve this, we introduce spatially-aware point-track embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the embedding similarity correlates directly with spatial proximity, enhancing the model's ability to distinguish and associate point-tracks. We inject these point-track embeddings into a video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/