培训后的过程如何塑造生物推理模型
How Post-Training Shapes Biological Reasoning Models
摘要
用于生物学的科学推理模型将语言模型与基于多模态生物数据（包括DNA、RNA和蛋白质）训练的基础模型相结合。这些模型是在训练之后才构建的，但各个训练阶段如何影响推理能力和泛化能力仍不清楚。我们研究了哪些时候后训练能提升性能，以及何时会导致过度专业化现象。在基因组学、转录组学和蛋白质领域，我们在不同训练方式下对100多个生物推理模型进行了训练和评估，这些方式包括主干模型调整、持续预训练、监督式微调以及强化学习等。我们同时测量了模型在域内和域外场景下的性能。我们发现，每个后训练阶段都会以不同的方式改变模型的泛化能力，而不是带来统一的提升效果。持续预训练能够通过将模型与生物语言相匹配来提升下游性能；监督式微调则能持续提升域内性能，但会导致域外性能提前达到峰值，随后随着模型适应训练分布而下降。当应用于具有匹配奖励机制的强监督式微调模型时，强化学习则能提升域外性能，并部分恢复模型的泛化能力。这些结果表明，生物推理的性能并非随着额外的监督或计算资源的增加而单调提升，而是取决于各训练阶段的组合方式。在固定的后训练预算下，最佳的域内-域外性能平衡取决于短暂的监督式微调、较大的强化学习投入，以及各训练阶段之间不对称适应能力。
English Abstract
Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are built through post-training, yet how each stage shapes reasoning and generalization remains poorly understood. We study when post-training improves performance and when it induces over-specialization. Across genomics, transcriptomics, and proteins, we train and evaluate more than 100 biological reasoning models under controlled variation in backbone, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL), measuring both in-domain (ID) and out-of-domain (OOD) performance. We find that each post-training stage reshapes generalization in a distinct way rather than contributing uniform gains. CPT improves downstream performance by aligning models with biological language. SFT consistently increases ID performance but causes OOD performance to peak early and decline as models fit the training distribution. RL, when applied to strong SFT checkpoints with aligned rewards, improves OOD performance and partially recovers generalization. These results show that biological reasoning does not improve monotonically with additional supervision or compute. Instead, performance depends on how training stages are composed. Under fixed post-training budgets, the strongest ID-OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity across stages.