揭开数据受限语言模型预训练中训练时间增强的神秘面纱
Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
摘要
随着人工智能实验室面临数据限制问题，计算能力超过了高质量文本生成的速率，语言模型预训练方式逐渐转向以数据量为约束、计算资源充足的模式，这种模式要求对固定数据集进行多次迭代的训练。在这种情况下，标准的自回归预训练方法会出现严重的过拟合现象，过早达到最优状态，之后性能还会持续下降。我们研究了训练过程中数据增强技术作为正则化手段，以缓解过拟合现象，从而能够在同一数据集上进行数百次迭代的有效训练。我们提出了三种不同的数据增强方法：令牌级噪声处理（如屏蔽、随机替换）、序列排列方式（从右到左预测、中间填充），以及目标偏移量预测（当i>1时，x_{t+i}）。通过系统的实验分析，我们发现这些增强方法能够延缓过拟合现象，降低验证损失。其中，随机令牌替换方式在各种方法中实现了最低的验证损失。结合多种增强方法则能进一步降低最小验证损失。我们的实验表明，数据增强技术能够解决数据受限情况下的预训练问题，为这一挑战提供了有前景的解决方案~\footnote{所有代码和数据均可在https://github.com/michaelchen-lab/data-augmentations-for-pretraining获取。}
English Abstract
As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate training-time data augmentation as a regularizer to mitigate this overfitting and enable productive training for hundreds of epochs on the same data. We introduce three orthogonal categories of augmentation for AR pretraining: token-level noise (masking, random replacement), sequence permutations (right-to-left prediction, Fill-in-the-Middle), and target offset prediction (x_{t+i} for i > 1). Through systematic ablations, we find that individual augmentations delay overfitting and lower validation loss relative to the baseline, with random token replacement achieving the best minimum loss among individual methods. Combining augmentation categories further lowers the minimum validation loss. Our experiments demonstrate that data augmentations mitigate AR pretraining's data inefficiency and offer a promising solution to the data-constrained regime~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.