Tmax：终端药物的一种简单制备方法
Tmax: A simple recipe for terminal agents
摘要
使用终端代理的模型已经迅速成为语言模型最流行的下游应用形式。尽管这种应用非常普遍，但关于基于强化学习的训练方法的学术研究却相对较少，这可能是因为相关的评估标准较为困难、数据不足，以及缺乏简单的训练方法。我们提出了Tmax——迄今为止最强大的开放式强化学习训练方法，使得开放式的数据使用方法能够更接近前沿水平。虽然这种方法相对简单，但仅使用90亿个参数就能在Terminal-Bench 2.0上获得27%的性能，远远优于之前那些参数更多的模型。具体来说，我们采用了一种新颖的分类方式来生成数据，结合了难度控制、不同角色设定以及多种验证方式，从而能够低成本地生成大量适用于强化学习和监督微调训练的终端环境数据。我们将这个终端数据集开源了，其规模是之前发布的终端代理数据集的2.5倍以上。然后，我们使用这些数据通过简单的强化学习方法来训练开放权重模型。我们在https://github.com/hamishivi/tmax上发布了我们的数据、模型和代码，作为未来关于终端代理研究的良好参考。
English Abstract
Terminal-using agents have quickly become the most popular downstream application of language models (LMs). Despite their prevalence, relatively little academic work has examined RL-based training of these models, likely due to difficult benchmarks, a lack of data, and a lack of simple baseline recipes. We present Tmax, the strongest open RL recipe for terminal agents to date, bringing open data recipes closer to the frontier. While simple, our recipe achieves 27\% on Terminal-Bench 2.0 with only 9B parameters, outperforming much larger models from prior work. Concretely, we generate data using a novel taxonomy, combining difficulty control, personas, and verifier diversification, which allows us to cheaply generate large amounts of terminal environments for RL and SFT training. We open-source our terminal dataset, which is over 2.5x larger than previously released terminal-agent datasets. We then train open-weight models using RL with our data, using a simple, outcome-only recipe. We release our data, models, and code as a strong baseline for future open academic work on terminal agents at https://github.com/hamishivi/tmax.