近似策略优化：以教师为主导的优化过程，而非通过梯度来优化策略
Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients
摘要
知识蒸馏技术将教师的智能转移给规模较小的学生。不过，这种方法的稳定性较差：因为学生需要模仿来自规模更大的教师的逻辑模型，这导致学生的能力主要集中在教师最擅长领域的技巧上，从而影响了其在训练数据之外的基准测试任务中的泛化能力。强化学习则通过让学生自己尝试不同的策略来避免这种模仿现象。然而，在那些每次尝试都失败的问题上——即学生得到的奖励为零，这些尝试被直接忽略——将更强大的教师反馈引入策略梯度中，会打破“基于策略的优化”假设，从而导致模型性能的下降。我们提出了“最近发展区优化”方法，该方法借鉴了维果茨基的“最近发展区”概念，将教师的智能融入到提示中，而不是单纯依赖于策略梯度。在处理困难问题时，ZPPO采用了两种重新设计的提示方式：一种是“二元候选问题”，它将一个正确的教师回答与一个错误的学生回答配对起来，作为匿名候选方案，让学生进行区分；另一种则是“负面候选问题”，它将学生所有的错误尝试汇总在一起，形成单一的提示，从而揭示出他们共同的失败模式。每个困难问题的提示都会反复使用，直到学生的表现达到一定的水平，或者因为容量限制而被淘汰。这样，ZPPO能够在学生当前的最近发展区内发挥最大的作用。在Qwen3.5框架下，针对四个不同规模的学生群体（0.8B到9B参数规模），以及27B参数的教师模型，经过训练后，ZPPO在31个基准测试任务中的表现优于其他基于策略或基于强化学习的算法，尤其是在最小规模的任务上取得了显著的进步。
English Abstract
Knowledge distillation transfers a teacher's competence to a small student but is brittle in the small-student regime: forcing the student to imitate logits from a much larger teacher concentrates it on the teacher's sharpest modes, hurting generalization on benchmark families beyond the training corpus. Reinforcement learning (RL) avoids logit imitation by training on the student's own rollouts. However, on questions where every rollout fails-yielding zero advantage and being silently discarded-injecting a stronger teacher's response into the policy gradient breaks the on-policy assumption and induces drift. We introduce Zone of Proximal Policy Optimization (ZPPO), inspired by Vygotsky's zone of proximal development, which keeps the teacher inside the prompt rather than the policy gradient. On hard questions, ZPPO constructs two reformulated prompts: a Binary Candidate-included Question (BCQ) pairs one correct teacher response with one incorrect student response as anonymized candidates the student must discriminate, and a Negative Candidate-included Question (NCQ) aggregates the student's wrong rollouts into a single prompt to surface their shared failure modes. A prompt replay buffer recirculates each hard question until it either graduates-the student's mean rollout accuracy on it reaches half- or is FIFO-evicted under finite capacity, amplifying BCQ and NCQ inside the student's current zone of proximal development. On the Qwen3.5 family at four student scales (0.8B-9B) with a 27B teacher, post-trained as vision-language models and evaluated on a 31-benchmark suite (16 VLM, 10 LLM, 5 Video), ZPPO outperforms off/on-policy distillation and GRPO, with the largest gains at the smallest scale.