MMDiff：扩展扩散变换器以用于多模态生成
MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
摘要
扩散变换器展现出了卓越的生成能力。然而，在去噪过程中生成的丰富感知表征在内容被渲染后会被丢弃。我们提出了MMDiff框架，该框架能够将冻结的扩散变换器转化为多模态生成系统，该系统能够结合各种感知特征来生成图像。我们的核心发现是：感知信息在去噪过程中是随时间分布的，而采用多时间步长的特征融合方式，并结合空间上不同的聚合权重，可以显著提高语义分割效果——相比单时间步长的提取方法，我们的结果在mIoU指标上提升了28.7%。此外，我们还采用了基于概念的注意力机制来进行空间引导，结果表明，冻结后的扩散特征能够与诸如DINOv3这样的先进编码器相媲美，甚至还能为其提供补充。通过仅在冻结的骨干网络上训练轻量级解码器，我们实现了在语义分割、显著物体检测以及深度估计方面的强大性能。这表明，这种框架能够在大规模场景下实现有效的合成数据生成。
English Abstract
Diffusion transformers have demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across their denoising trajectory are discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into a multi-modal generative system that jointly produces images alongside any combination of dense perceptual modalities using lightweight decoder heads. Our central finding is that perceptual information is temporally distributed along the denoising trajectory, and that multi-timestep feature fusion with spatially varying aggregation weights is essential, improving semantic segmentation results by up to 28.7% mIoU over single-timestep extraction. We further adopt concept-driven attention extraction for interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such as DINOv3. By training only lightweight decoder heads on a frozen backbone, we achieve strong performance in semantic segmentation, salient object detection, and depth estimation, and demonstrate that this framework enables effective synthetic data generation at scale.