LISA：用于可控制生成效果的视觉条件概率评分对齐
LISA: Likelihood Score Alignment for Visual-condition Controllable Generation
摘要
目前流行的双分支架构模式——即训练一个辅助网络来编码视觉特征，并将其中间层的特征与预训练的主网络结合——在实现可控制的视觉生成任务中取得了显著成效。尽管这种架构被广泛采用，但辅助网络的作用及其训练效率仍未被充分研究。本文从基于得分的生成建模角度重新审视这一主流架构：1）主网络通过提供无条件的得分信息来保持视觉感知质量；2）辅助网络则通过隐含地提供似然得分来实现条件控制。基于这一视角，我们提出了LIkelihood Score Alignment（LISA）方法，这是一种有效的正则化手段，能够明确地将辅助网络的中间特征与近似的似然得分进行匹配。具体来说，我们首先从辅助网络的特定层提取特征，然后通过轻量级解码器将这些特征映射到得分潜在空间；接着，我们构建出一个近似的似然得分目标，并计算解码器输出与该目标之间的距离作为额外的正则化损失。最后，我们同时利用标准扩散损失和我们的正则化损失来优化辅助网络和解码器。在多种图像/视频任务、不同架构以及不同扩散/流模型下的实验表明，LISA不仅能够有效加速训练收敛过程并提升最终生成的图像质量，还能使辅助网络的各特征更加独立，从而简化条件化建模过程，同时减少额外的训练成本和推理成本。
English Abstract
The prevalent dual-branch paradigm, i.e., training a side network to encode visual conditions and fusing its intermediate-layer features to a frozen pretrained main network, has shown remarkable success in visual-condition controllable generation. Despite its widespread adoption, the role of the side branch and its training efficiency remain underexplored. In this paper, we first revisit this mainstream paradigm through the lens of score-based generative modeling: 1) The main network preserves visual perceptual quality by providing a prior unconditional score. 2) The side network steers conditional control by implicitly contributing a likelihood score. Guided by this perspective, we propose LIkelihood Score Alignment (LISA), an effective regularization method that explicitly aligns the intermediate feature of the side network with an approximated likelihood score. Specifically, we first hook features from a designated layer of the side network and project them into the score latent space by a lightweight decoder. Then, we construct an approximated likelihood score target and calculate the distance between the decoder's output and this target as an additional regularization loss. Finally, we jointly optimize the side network and decoder with both standard diffusion loss and our regularization loss. Experiments across various image/video tasks, architectures, and diffusion/flow models demonstrated that LISA can not only consistently accelerate the training convergence and improve final synthetic results, but also encourage the side network's features to be more disentangled for conditional modeling with negligible additional training cost and zero extra inference cost.