从“自我未来”中学习：适用于多语言大语言模型的自策略自融合方法
Learning from the Self-future: On-policy Self-distillation for dLLMs
摘要
在训练后的大型语言模型上，基于策略的自精炼方法已被证明非常有效。不过，这种方法来处理扩散型语言模型的情况则尚未得到充分探索。现有的自精炼方法本质上都是以自回归机制为基础的；它们通过从左到右的前缀条件化以及令牌级别的差异监督来注入“特权信息”，但这种设计与扩散型语言模型的任意顺序生成方式存在根本冲突。我们提出了d-OPSD这一专为扩散型语言模型设计的自精炼方法。我们的方法有两个核心贡献：首先，我们通过使用模型自身生成的答案作为后缀条件，重新定义了自教师构建的方式，使得学生模型能够从“自身的未来经验”中学习，而不是依赖特权前缀的信息。其次，我们将监督机制从令牌级别转移到了步骤级别，使得训练过程能够与扩散型语言模型的迭代去噪过程相契合。在四个推理基准测试中的实验表明，d-OPSD在样本效率方面始终优于RLVR和SFT等基线方法，其所需的优化步骤仅相当于RLVR的10%左右，这为扩散型语言模型的后续训练开辟了一条有前景的路径。代码可在https://github.com/xingzhejun/d-OPSD下载。
English Abstract
On-policy self-distillation (OPSD) has proven effective for post-training large language models (LLMs), yet its application to diffusion LLMs (dLLMs) remains unexplored. Existing OPSD methods are inherently autoregressive-centric. They inject privileged information via left-to-right prefix conditioning with token-level divergence supervision, a design that fundamentally conflicts with the arbitraryorder generation of dLLMs. We introduce d-OPSD, the first OPSD framework tailored for dLLMs. Our approach makes two core contributions. First, we reframe self-teacher construction by using self-generated answers as suffix conditioning, enabling the student model to learn from "self future-experience" rather than privileged prefixes. Second, we shift supervision from token-level to step-level, aligning training with the iterative denoising process of dLLMs. Experiments across four reasoning benchmarks show that d-OPSD consistently outperforms RLVR and SFT baselines with superior sample efficiency, requiring only around 10% of the optimization steps by RLVR and opening a promising pathway for dLLM posttraining. The code is available at https://github.com/xingzhejun/d-OPSD.