ProMSA：用于基于知识的视觉问答的渐进式多模式搜索代理
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
摘要
基于知识的视觉问答系统需要模型将图像信息与外部知识相结合。大多数现有方法采用固定的检索-生成流程，使用预先选定的检索器以及固定的top-k设置，这种方式在推理过程中无法灵活调整。我们提出了ProMSA这一用于KB-VQA的多模态渐进式搜索算法。对于每一对图像与问题对，该算法会依次选择图像检索、文本检索或停止操作，同时遵循明确的工具调用限制，并避免重复检索。在训练过程中，我们首先使用拒绝采样SFT来学习有效的工具使用方式，然后利用TN-GSPO进行优化——这是一种序列级强化学习方法，它根据生成长度和工具交互深度对更新进行归一化处理。在E-VQA和InfoSeek上的实验表明，ProMSA相比其他强大的RAG系统和算法基线具有显著的性能提升，同时也提高了检索效率和端到端识别准确率。代码可在https://github.com/DingWu1021/Promsa获取。
English Abstract
Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.