MVEB：大规模视频嵌入基准测试
MVEB: Massive Video Embedding Benchmark
摘要
我们介绍了Massive Video Embedding Benchmark（MVEB），这是一个包含23个任务的视频嵌入基准测试，涵盖了分类、零样本分类、聚类、成对分类、检索以及以视频为中心的问题回答等场景。我们对33种模型进行了评估，发现没有任何一种模型能够占据主导地位：基于MLL的嵌入在分类、聚类、成对分类和问答任务中表现最佳；而多模态绑定方法则在检索和零样本分类任务中表现出色；而那些没有使用对比度适应的生成式MLL模型则在跨模态任务中表现较差。对仅视频或音频+视频数据的评估表明，音频的贡献程度取决于数据集的标注来源——当标签来自两种模态时，音频能发挥积极作用；而当标签仅来自视觉数据时，音频的作用则相反。这种差异在多种模型家族中都有体现，差距达到了6点。MVEB是从MVEB+衍生出来的，后者是一个包含184个任务的基准测试集。该基准测试旨在在保持任务多样性的同时，降低评估成本。它可以被整合到MTEB生态系统中，从而实现文本、图像、音频和视频之间的统一评估。我们将MVEB以及所有184个任务的相关代码和排行榜发布在https://github.com/embeddings-benchmark/mteb网站上。
English Abstract
We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering. We evaluate 33 models and find that no single model dominates: MLLM-based embeddings lead on classification, clustering, pair classification, and QA; multimodal binding leads on retrieval and zero-shot classification; generative MLLMs without contrastive adaptation collapse on cross-modal tasks. Paired video-only vs. audio+video evaluations show that audio's contribution depends on dataset annotation provenance: audio helps when labels were produced from both modalities and hurts when they were produced from visuals alone, a six-point gap consistent across model families. MVEB is derived from MVEB+, a 184-task pool, and is designed to maintain task diversity while reducing evaluation cost. It integrates into the MTEB ecosystem for unified evaluation across text, image, audio, and video. We release MVEB and all 184 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.