Sumi：从零开始构建统一扩散语言模型
Sumi: Open Uniform Diffusion Language Model from Scratch
摘要
扩散模型已成为替代自回归模型的很有前景的选择。其中，均匀扩散语言模型允许任何token在任何步骤中被更新，从而能够实现更灵活的生成方式。不过，目前还没有任何均匀扩散语言模型能够在大规模参数和大量token的约束下被从头开始预训练。无论是自回归建模还是掩码扩散建模，都已经存在能够用于研究的模型；而均匀扩散模型则尚未有这样的模型。一个经过大规模预训练的均匀扩散语言模型可以为研究其规模效应、生成机制、可控制性以及与现有自回归和掩码扩散模型之间的权衡关系提供清晰的参考点。为此，我们推出了Sumi——一个完全开放的7B参数均匀扩散语言模型，该模型是在1.5T个token上从头开始预训练的。在知识、推理和编程等评估指标上，Sumi的表现与那些使用相同数量token进行训练的自回归模型相当；但在常识相关评估指标上则表现较差，这很可能是因为我们的数据中包含较多教育相关的内容。我们公开了模型的权重、检查点以及完整的训练流程，包括关于数据混合方式的完整说明。我们希望这次发布能够帮助社区深入研究大规模均匀扩散模型，并推动对其尚未被充分理解的方面进行研究。
English Abstract
Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.