相信合适的老师:注重质量的自我蒸馏法用于GUI grounded
Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
摘要
图形用户界面(GUI)的识别任务需要视觉-语言模型能够从高分辨率屏幕截图中识别出小的目标元素,并准确预测出屏幕上的坐标位置。基于策略的自去化方法是一种有效的后训练方法,因为它能够提供丰富的标记级信号,而不仅仅是难以确定的坐标信息。然而,简单的基于策略的自去化方法并不适合用于GUI识别任务:该方法会针对学生生成的预测结果来评估教师信号的质量,但当预测结果偏离了真实坐标时,教师信号的质量就会下降,从而导致不可靠的信号。为了解决这一问题,我们提出了一种注重质量的自去化方法,通过软性正确性检测机制以及教师概率的缩放机制来提升坐标标记信号的质量。软性正确性检测机制可以判断当前教师对坐标标记的预测是否仍然符合学生生成的预测结果;如果不符合,那么相应的教师信号就会被降低权重。而教师概率的缩放则利用教师的置信度作为调整信号强度的因素。实验结果表明,单独使用这两种机制都无法提升整体性能,而将它们结合起来则能显著改善性能。这表明这两种机制可以互补:正确性检测机制可以排除不可靠的标记信号,而教师概率的缩放则可以进一步调整剩余信号的强度。在六个GUI识别基准测试中的实验表明,我们的方法能够持续提升基础模型的性能,并且优于其他优秀基线模型。
English Abstract
Graphical user interface (GUI) grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task, since it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well suited to GUI grounding: OPSD evaluates the teacher on student-generated prefixes, the quality of coordinate-token teacher signals can degrade when the prefix has already deviated from the target coordinate, leading to unreliable teacher signal. To mitigate this, We propose quality-aware self-distillation for VLM-based GUI grounding, which improves coordinate-token teacher-signal quality through soft correctness-aware gating and teacher-probability scaling. The soft correctness-aware gate checks whether the teacher's current coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix. If not, the corresponding teacher signal is down-weighted. Teacher-probability scaling then uses the teacher's confidence as a lightweight factor to further calibrate the strength of the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently improves performance. This suggests that the two mechanisms play complementary roles: correctness-aware gating suppresses unreliable coordinate-token supervision, while teacher-probability scaling calibrates the strength of the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.