VisualClaw：一种适用于物理世界的实时、个性化智能代理
VisualClaw: A Real-Time, Personalized Agent for the Physical World
摘要
视觉语言模型可以作为处理复杂多模态任务的通用接口。不过，其应用仍然存在三个问题：首先，视觉语言模型在处理密集的视频帧和长文本提示时，通常会产生较高的延迟和成本；其次，模型在部署后仍然保持静态状态；最后，现有的视频问答基准测试并未涵盖模型在工具使用场景中能否利用视觉信息的问题。我们提出了VisualClaw这一自进化型多模态智能体，其设计基于两个核心原则。首先，混合编码方式通过级联过滤机制来减少不必要的信息流，同时通过对文本技能库进行压缩处理来降低成本。其次，智能体能够从失败中学习：通过检索到的记忆信息来指导技能的学习过程，从而生成更有用的技能更新，帮助模型更好地应对未来的问题。在4个视频问答基准测试中，使用VisualClaw的模型平均可以将每个问题的API调用次数减少98%，而离线8帧基准测试则减少了25.9%。此外，VisualClaw在大多数场景下的准确率也有所提升，例如在EgoSchema测试上，使用VisualClaw的模型准确率提升了3.85%，而在Gemini 3 Flash测试上则提升了15.80%。为了弥补这些不足，我们开发了VisualClawArena——一个包含200种场景的多模态智能体基准测试平台。在这个平台上，模型需要在工作空间中同时使用视频信息、文档数据、动态更新以及可执行检查等元素。在VisualClawArena测试中，使用相同框架且配备计算机辅助智能体的模型，其准确率分别比不使用该框架的模型高出2.9%（针对Codex（GPT-5.5））和3.2%（针对Claude Code（Sonnet 4.6））。与均匀采样基准测试相比，使用VisualClawArena的模型的成本降低了9.5%。这些特性使得VisualClaw非常适合边缘应用场景——在这些场景中，级联处理机制可以将1小时的视频流信息处理简化为5到20次调用，而自进化功能则使其成为完美的个性化助手。
English Abstract
Vision language models are serving as general-purpose interfaces for complex multimodal tasks. However, deployment still faces three gaps: VLMs typically incur high latency and cost when processing dense video frames and long prompts, the agent scaffold remains static after deployment, and standard video-QA benchmarks do not test whether agents can use visual evidence inside tool-using workspaces. We present VisualClaw, a self-evolving multimodal agent built around two principles. First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate and compressing the text skill bank through hot/cold top-k injection. Second, skill evolution lets the agent learn from failures: retrieved memories condition an evolver as direct concatenated context or as guided evidence, producing skill-bank updates that help future questions. Across 4 video-QA benchmarks with 2 VLMs, VisualClaw cuts per-question API cost by an average -98% versus full-frame upload and by -25.9% over the offline uniform 8 frame baseline, while boosting accuracy in most settings, e.g., an average +3.85% and a peak +15.80% on EgoSchema with Gemini 3 Flash. To address the gap, we curate VisualClawArena, a 200-scenario multimodal agentic benchmark built through a strict five-stage pipeline; models must use video evidence, documents, dynamic updates, and executable checks inside a workspace. On VisualClawArena, the same framework with computer-use agent backends improves macro accuracy by +2.9% for Codex (GPT-5.5) and +3.2% for Claude Code (Sonnet 4.6) over no-evolution baselines, with a -9.5% cost reduction compared to the uniform-sampled baseline. These properties make VisualClaw a natural fit for edge applications, where the cascade reduces a 1-hour streaming session from ~3,600 API uploads down to only 5-20 calls and the self-evolution makes it a perfect personalized assistant.