‹ 返回 2026-06-15

ProCUA-SFT Technical Report

ProCUA-SFT Technical Report

▲ 1 NVIDIA、University of Washington · 机器学习、计算机视觉、多模态大模型、智能体、桌面自动化 提交者 NVIDIA、University of Washington 2026-06-15

Jaehun Jung, Ximing Lu, Brandon Cui, Muhammad Khalifa, Shaokun Zhang, Hao Zhang, Jin Xu, Amala Sanjay Deshmukh, Karan Sapra, Andrew Tao, Yejin Choi, Jan Kautz, Mingjie Liu, Yi Dong

摘要

ProCUA-SFT是一个包含310万步级监督微调(SFT)样本的大规模开源数据集,旨在训练计算机使用智能体(CUAs)。该数据集通过全自动流水线生成,包含9.3万条完全合成的桌面操作轨迹,覆盖2484种应用程序组合。其核心设计包括基于真实环境状态的任务合成、复杂真实文档的内容注入以及单视觉语言模型(VLM)的闭环验证,有效解决了现有数据集在微调时导致的负迁移问题,显著提升了智能体在复杂桌面任务中的执行能力。

English Abstract

Training computer-use agents (CUAs) -- models that interact with graphical desktops through screenshots and keyboard/mouse actions -- requires large-scale, diverse trajectory data collected in full desktop environments. The largest public resource, AgentNet (22.5K human trajectories), leads to negative transfer when used for supervised fine-tuning (SFT): continuing training UI-TARS 7B on AgentNet causes OSWorld success rate to fall from 26.3% to 8-10%. We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories across 2,484 application combinations. The dataset is produced by a fully automated pipeline that (i) synthesizes grounded tasks on live desktops seeded with real-world content -- 912 spreadsheets from SpreadsheetBench, approximately 10K permissively-licensed presentations from Zenodo10K, and multi-application OSWorld configs -- and (ii) verifies each task's feasibility through binary precondition checking before rollout. A single VLM (Kimi-K2.5) serves as goal generator, precondition judge, and trajectory executor, eliminating planner-actor capability gaps. Each trajectory is expanded into step-prefix samples that exactly reproduce the context layout seen at inference time. Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld -- an 18.7 percentage-point improvement over the base model and over 35% above AgentNet-trained counterparts. A subset of ProCUA was incorporated into the training data for the Nemotron 3 Nano Omni model, contributing to its computer-use capabilities.