DiffusionBench：关于扩散Transformer的全面评估
DiffusionBench: On Holistic Evaluation of Diffusion Transformers
摘要
在图像生成领域，扩散变换器（DiT）的研究最终都集中在同一个评估框架上：即基于ImageNet的类别条件生成任务。虽然这些方法能够提升FID等相关指标，但很难确定它们是否真的代表了生成模型领域的实质性进步。另一种自然的选择——文本到图像生成——则因为训练和评估成本过高或操作不便而常被忽略。我们认为这种看法已经不再成立。我们提出了NanoGen这一统一的DiT训练与评估框架。NanoGen能够在ImageNet上达到最先进的DiT基准水平，同时通过12种配置调整方式，还能训练出具有竞争力的文本到图像模型。目前，NanoGen支持RAE、VAE、像素空间以及MeanFlow等扩散方法，且适用于ImageNet和文本到图像两种场景。在NanoGen框架下，训练文本到图像模型所需的计算资源与训练ImageNet模型相当。通过用NanoGen训练21个潜在扩散模型后，我们发现，不同方法的性能排名与ImageNet和文本到图像生成任务之间并没有明显的关联：三个指标的皮尔逊相关系数介于-0.377到-0.580之间。这表明，一种能够提升ImageNet上类别条件生成的FID指标的方法，可能在文本到图像生成任务中并不会带来相应的改善，这充分说明有必要对DiT在两种任务上的表现进行评估。为此，我们汇总了ImageNet和文本到图像生成任务的实验结果，从而形成了DiffusionBench这一用于评估DiT研究的综合基准。我们建议以DiffusionBench作为替代指标来报告研究成果，因为能够提升DiffusionBench指标的方法，更有可能反映出更广泛的进步。
English Abstract
Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While methods improve the FID and related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I setups. Under NanoGen, training T2I requires comparable compute to ImageNet. After training 21 latent diffusion models with NanoGen, we observe that method ranking shows no strong correlation between ImageNet and T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditional ImageNet FID may show no corresponding improvement on T2I, clearly indicating the necessity of evaluating DiTs on both tasks. To this end, we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research. We recommend reporting DiffusionBench in place of ImageNet alone: methods that improve DiffusionBench are more likely to reflect broader progress.