PhoneHarness：通过混合式GUI、命令行界面以及工具操作来优化手机使用体验
PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions
摘要
人们越来越期望手机代理能够完成真正的移动工作流程，而不仅仅是预测下一个屏幕操作。不过，目前大多数关于移动代理的研究仍然主要将代理视为仅仅负责处理用户界面操作的工具——它们只能观察屏幕上的信息，执行点击、滑动等操作，而最终的评估则基于目标应用程序的状态。但实际上，手机使用中的任务要复杂得多：用户需要决定何时使用应用程序的界面操作、设备端的命令，还是结构化工具来完成任务；同时，还需要确保预期的效果真正实现了。我们提出了PhoneHarness这一框架，它是一个用于测试手机代理在可验证的移动工作流程中表现的工具。PhoneHarness允许在GUI、CLI以及主机端的工具操作之间实现联动，既保证了操作的确定性，又实现了有约束条件的界面委托机制，同时还能生成可审核的执行日志。其测试基准测试名为PhoneHarness Bench，该测试旨在评估代理是否能够完成具有可观察效果的任务，而不仅仅是能否给出合理的最终结果。在经过注释后的评估中，PhoneHarness的通过率达到了75.0%，比最强大的非PhoneHarness方案高出12.9个百分点。因此，PhoneHarness和PhoneHarness Bench分别扮演着不同但相互依赖的角色：前者使复杂的手机工作流程变得可实施，而后者则衡量代理是否能够可靠且安全地使用这种框架。我们的研究结果表明，可靠的手机自动化依赖于操作的确定性以及可验证的执行过程，而不仅仅是视觉化的界面控制。
English Abstract
Phone agents are increasingly expected to complete real mobile workflows rather than merely predict the next screen action. However, much of the current mobile-agent literature still evaluates agents primarily as GUI controllers that observe a screen, emit taps and swipes, and are scored by target app state. Real phone-use tasks are broader: they require deciding when to use app GUIs, device-side commands, or structured tools, while leaving evidence that the intended side effect actually occurred. We introduce PhoneHarness, a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile workflows. PhoneHarness runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. PhoneHarness and PhoneHarness Bench therefore play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.