UniDDT：通过分离扩散变换技术，实现多模态理解与生成的统一处理
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
摘要
统一多模态模型（Unified Multimodal Models，简称UMMs）已成为实现通用多模态智能的关键方向。这些模型将理解与生成功能整合到一个统一的框架中。然而，现有的UMMs仍面临一些主要挑战：首先，视觉理解与生成任务之间存在着固有的学习冲突，这导致这两个任务的建模效果都不理想；其次，不同的理解与生成视觉空间模式会阻碍模型的扩展性；最后，当前的方法过于依赖特定任务的数据，而忽视了文本与图像理解及生成的双重性特征。为了应对这些挑战，我们提出了UniDDT模型。该模型利用带有噪声处理的ViT编码器以及大型语言模型来统一视觉生成与理解任务中的语义编码过程，同时采用独立的扩散解码器来分离扩散解码过程与文本解码过程。通过这种结构，UniDDT能够将一个潜在空间作为统一的视觉表示，从而实现理解与生成任务的无缝结合。这样一来，生成任务的扩展性与理解任务的语义表达能力就能得到平衡。此外，我们还从相同的图像-文本对中提取出两种不同的数据结构，从而加强生成数据与理解数据之间的相互依赖性，进一步发挥它们的双重特性。广泛的实验表明，UniDDT在多模态理解与生成方面实现了有效的统一，同时提升了语义一致性和扩展性。在视觉生成任务中，UniDDT的GenEval得分达到了0.87，整体得分则达到了86.9；而在多模态理解任务中，UniDDT在MME基准测试中的得分达到了1699.5，整体得分则为76.5。
English Abstract
Unified Multimodal Models (UMMs) have emerged as a critical direction for general-purpose multimodal intelligence, integrating understanding and generation into a single framework. However, existing UMMs face prominent challenges: (1) the inherent learning conflicts between visual understanding and generation tasks, leading to suboptimal modeling in both tasks; (2) different understanding and generation visual spaces impeding scalability; (3) over-reliance on task-specific data that neglects the duality of text-image understanding and generation. To address these challenges, we propose UniDDT, which leverages a Noisy ViT encoder along with an LLM to unify semantic encoding for visual generation and understanding tasks, while employing a separate diffusion decoder to decouple diffusion decoding from text decoding. With this Noisy ViT encoder, UniDDT is able to leverage the latent space as a unified visual representation, enabling seamless compatibility between understanding and generation tasks. Thus, the scalability within the generation tasks and the semantic expressiveness within understanding tasks can be balanced. Also, we construct dual data structures from the same image-text pairs, fostering interdependence between the generation and understanding data to exploit their inherent duality. Extensive experiments demonstrate that UniDDT achieves effective unification of multimodal understanding and generation with enhanced semantic consistency and scalability. For visual generation tasks, our UniDDT achieves 0.87 GenEval score and 86.9 DPG overall score. For multimodal understanding tasks, our UniDDT achieves 1699.5 score on MME benchmark and 76.5 overall score on SEEDbench.