Multi-LCB:将LiveCodeBench扩展至多种编程语言
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
摘要
LiveCodeBench (LCB) 最近成为评估大型语言模型在代码生成任务中的性能的常用基准测试工具。通过收集各种编程挑战题、不断添加新的题目,并根据发布时间对题目进行筛选,LCB能够确保评估的可靠性,同时提供对编码能力的全面了解。不过,LCB仅适用于Python语言,因此尚不清楚大型语言模型是否能够在现实世界软件工程中所需的多种编程语言中表现出良好的性能。 我们提出了Multi-LCB这一基准测试工具,它可以在十二种编程语言中进行大型语言模型的评估,包括Python。Multi-LCB能够将LCB数据集中的Python相关任务转换为其他语言的等效任务,同时保留LCB的评估机制。由于与原始LCB格式完全兼容,Multi-LCB可以自动跟踪未来的LCB更新,从而能够系统地评估不同语言下的代码生成能力,让模型在超出Python语言范围的情况下也能保持良好的性能。 我们在Multi-LCB上对24种大型语言模型进行了指令处理与推理能力的测试,发现存在Python过度拟合现象、语言特定的问题以及多语言性能上的巨大差异。我们的研究结果表明,Multi-LCB是一种有效的多编程语言代码评估基准,能够有效解决LCB的主要缺陷,并揭示当前大型语言模型存在的关键不足。
English Abstract
LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.