CLI-Universe：面向终端代理的可验证任务合成引擎
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
摘要
虽然基于大语言模型的终端代理在最近已经展现出了令人期待的能力，但高质量、可执行的训练数据匮乏仍然是一个严重的障碍。现有的合成流程通常是通过将表面层面的信息融入到任务中来实现，这往往会导致指令模糊、执行路径有限以及测试结果不稳定，从而无法提供有效的学习信号。为了克服这一问题，我们提出了CLI-Universe这一合成引擎，它能够生成适合终端代理的任务。CLI-Universe通过从多维能力分类体系（如领域、技能类型、能力以及技术层面）中选择组合来生成候选任务，然后通过对真实世界技术资料进行深入研究来验证这些候选任务。为了确保结果的可靠性，这些候选任务会被转化为Docker化环境，并经过多阶段的验证流程：包括基于评分标准的测试设计、条件性过滤机制以及严格的失败检测机制。在整个验证过程中，大约三分之二的候选任务会被排除，最终只保留那些真正有效且并非过于复杂的任务。为了验证我们的框架，我们创建了一个包含6,000条轨迹的数据集，称为CLI-Universe-6K。有趣的是，对Qwen3-32B模型进行微调后，其在Terminal-Bench 2.0上的性能达到了33.4%。这一成绩使得在开源数据上训练的模型在32B参数以下时达到了新的最优水平，其性能甚至超过了那些参数更多的模型，充分证明了结构化、高保真度的合成方法在数据利用方面的优势。
English Abstract
While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principled synthesis engine that constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensional capability taxonomy (domain, skill type, capability, and engineering pillar), then grounds each candidate through evidence-guided deep research over real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated into Dockerized environments and subjected to a multi-stage executable verification pipeline featuring rubric-gated test construction, hint-conditional filtering, and strict fail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories called CLI-Universe-6K. Remarkably, fine-tuning Qwen3-32B on CLI-Universe-6K achieves 33.4% on Terminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis.