基于层次结构的优势权重分配方法，用于在线强化学习中对来自稀疏片段数据的VLAs进行微调
Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
摘要
当经过预训练的VLA策略通过在线强化学习进行微调时，每次实验只会产生一个二元结果——成功或失败。然而，智能体的更新仍然需要针对每个阶段进行监控。现有的方法通常试图将这种稀疏的结果转化为一个单一的标量奖励或优势信号，但这会混淆不同阶段的反馈信息，从而在基本任务成功之后无法提供有效的指导。首先，单一的标量信号使得“可行性”和“效率”这两个目标难以区分；一旦基本任务成功，二元标签就无法区分那些高效完成的任务与那些效率较低的任务。其次，现实中的实验过程包含自主执行阶段和干预阶段；如果简单地为这些阶段分配不同的结果标签，会导致错误的责任分配。为了解决这些问题，我们提出了分层优势加权行为克隆算法。该算法为这两个目标分别训练独立的评估模型，并将它们的输出结果结合起来，同时采用状态自适应机制来平衡这两种目标。状态自适应的门机制能够结合两者的优势信息：在可行性不确定的情况下优先考虑可行性，而在可行性较高时则专注于效率问题。最终，这一机制可以将结果转化为每个阶段对应的权重，从而用于智能体损失的计算。通过这种机制，我们可以确保结果标签仅适用于当前策略所执行的阶段，从而避免监督信息的泄露。在三项涉及身体互动的任务中，使用HABC算法的实验结果显示，成功率分别达到了92%、88%和38%，而使用传统微调方法的成功率仅为36%、44%和12%。
English Abstract
When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-transition supervision. Existing approaches commonly reduce this sparse outcome to a single scalar reward or advantage signal, which conflates distinct forms of transition-level feedback and provides limited guidance once basic task success becomes achievable. First, a single scalar signal conflates the two objectives of viability and efficiency; once basic success is achieved, the binary label provides no gradient to distinguish efficient completions from slow ones. Second, real-world rollouts mix autonomous and intervention segments; naively assigning episode outcomes across these boundaries introduces incorrect credit assignment. To address these issues, we propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for these two objectives on different data subsets and combines their outputs with a state-adaptive balance. A state-adaptive gate g_t merges their one-step advantages, prioritizing viability when success is uncertain and shifting to efficiency only when viability is high, and converts the result into per-transition weights on the actor loss. Intervention-aware credit assignment further restricts outcome labels to segments executed by the current policy, preventing supervision from leaking across intervention boundaries. In real-robot experiments on three contact-rich bimanual tasks, HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.