流形强盗：基于大语言模型潜在几何特性的贝叶斯课程学习
Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models
摘要
用于提升大语言模型推理能力的强化学习方法，可以通过贝叶斯流形课程框架得到改进。该框架根据任务流形关系以及内在的非平稳性来安排问题采样过程。这种强化学习方法由Qwen/Qwen2.5-Coder-32B-Instruct实现。强化学习是提升大语言模型推理能力的关键手段，而训练效率则取决于在优化过程中如何选择问题。现有的自适应课程学习方法通常倾向于选择中等难度的问题，将问题选择视为一个具有独立实验臂的随机实验问题，从而忽视了任务空间的结构化、异质性特征。在本研究中，我们将问题采样视为一种具有内在非平稳性的流形结构随机实验问题：各个问题通过模型的潜在表示空间相互关联，而采样决策可以影响学习信号在该空间中的演化方式。为了实现这一理念，我们提出了贝叶斯流形课程框架，该框架能够将问题组织成层次化的任务树结构，并运用贝叶斯学习方法来指导问题采样过程。实验表明，不同的采样策略会导致生产力（学习信号）、多样性（任务流形的覆盖程度）和实用性（评估的相关性）之间的复杂权衡。这些结果表明，仅关注问题的难度是不够的，必须同时考虑问题的结构和类型特征，才能取得良好的性能。
English Abstract
Reinforcement learning approaches for improving LLM reasoning capabilities are enhanced by a Bayesian Manifold Curriculum framework that structures problem sampling based on task manifold relationships and endogenous non-stationarity. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Reinforcement learning (RL) is a central approach for improving reasoning capabilities in large language models (LLMs), where training efficiency depends critically on how problems are sampled during optimization. Existing adaptive curriculum learning methods typically prioritize prompts of intermediate difficulty, treating problem selection as a standard bandit problem with independent arms and overlooking the structured, heterogeneous nature of the task space. In this work, we frame problem sampling as a manifold-structured bandit problem with endogenous non-stationarity : problems are related through the model's latent representation space , and sampling decisions can steer how learning signals evolve across that space. To operationalize this perspective, we introduce Bayesian Manifold Curriculum (BMC), a structure-aware framework that organizes problems into a hierarchical task tree and applies Bayesian learning to guide sampling. Empirically, we find that different sampling strategies induce non-trivial tradeoffs between productivity (learning signal), diversity (coverage of the task manifold), and utility (evaluation relevance). These results show that prioritizing difficulty alone is insufficient for strong downstream performance, highlighting the importance of incorporating structure and type-awareness into problem sampling .