VideoRAE：通过表示自编码器技术驯服视频基础模型以进行生成式建模
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
摘要
视频生成模型通常依赖由3D变分自编码器所学习的潜在空间。然而，传统的3D-VAE主要适用于像素级重建任务，这可能会限制其潜在空间在语义和时空结构方面的表现。与此同时，像V-JEPA 2和VideoMAEv2这样的视频基础模型具有强大的视频理解能力，但那些被冻结下来的表示能否被转化为紧凑、具备重建能力的视频潜在空间，目前仍未被充分研究。我们通过VideoRAE解决了这个问题——这是一种表示自编码器，它利用从冻结的视频基础编码器中得到的多尺度层次化特征，并通过轻量级的1D自注意力投影器对这些特征进行压缩处理。VideoRAE既支持用于扩散变换器的连续潜在空间，也支持用于自回归模型的离散标记，通过多编码簿高维量化技术实现。在解码过程中，与冻结的VFM教师模型之间的局部与全局表示对齐目标有助于保持语义信息，同时无需使用KL正则化即可完成训练。实验表明，VideoRAE在连续和离散两种情况下都能实现良好的重建效果。在UCF-101数据集上，它与AR和DiT生成器分别获得了40和93的最优类到视频gFVD值，其收敛速度也比其他自编码器基线模型快约5倍。在2B规模的文本到视频研究中，用VideoRAE替换LTX-VAE后，模型收敛速度更快，且各项参数设置保持不变。这些结果证明了冻结后的VFM表示可以作为灵活且适合生成的视频潜在空间。该模型和代码可访问于https://zhxie0117.github.io/VideoRAE。
English Abstract
Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.