没有资源，没有参照标准，就没有问题吗？在无资源语言中评估与改进用于代码生成的LLM
No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages
摘要
大型语言模型已经显著提升了软件工程任务的自动化水平。一个典型的例子就是代码生成——大型语言模型可以根据自然语言描述来生成特定编程语言的代码。目前，该领域的研究大多集中在资源丰富的语言上，比如Python或Java，因为这些语言拥有大量的训练数据。而针对资源匮乏的语言的研究则相对较少，因为这些语言在训练数据中占比很低。相反，那些几乎没有任何训练数据的语言则几乎未被研究过。这类语言通常出现在工业领域，企业会开发自有的、特定领域的语言，而这些语言无法被像GitHub Copilot这样的商业工具支持。因此，企业不得不自行开发代码推荐系统。为了探索可能的解决方案，我们构建了三个适用于资源匮乏语言的代码生成基准测试，这些语言所使用的两种编程语言几乎没有训练数据。通过这些基准测试，我们尝试了多种方法来让大型语言模型了解这些语言，包括基于提示的技术，以及利用有限数据进行的预训练和微调方法。虽然进一步的预训练能显著提升资源匮乏语言的性能，但直接将其应用于经过指令调整的模型会损害其遵循指令的能力。为了解决这个问题，我们从一个基础模型开始，然后在目标语言上进行进一步预训练，最后通过从指令模型转移权重来赋予模型遵循指令的能力。这种方法能够显著提升资源匮乏环境下的代码生成能力，使得企业能够以较低的成本部署专门的指令模型，而无需处理指令微调所需的计算成本。
English Abstract
Large Language Models (LLMs) have significantly advanced the automation of software engineering tasks. One prominent example is code generation, where an LLM produces code in a specified programming language based on a natural language description. Most research in this area has focused on high-resource languages, such as Python or Java, which benefit from abundant training data. A smaller body of work has explored low-resource languages, which are underrepresented in training corpora. In contrast, no-resource languages for which LLMs have seen virtually no training data remain largely unstudied. These languages often emerge in industry, where organizations develop proprietary or domain-specific languages unsupported by commercial tools like GitHub Copilot. This results in the need for companies to deploy their own in-house code recommenders. To investigate possible solutions in this context, we build and release three code generation benchmarks for no-resource languages, based on two recently proposed programming languages for which very little training data is available. Using these benchmarks, we experiment several solutions to teach LLMs about no-resource languages, including prompt-based techniques as well as pre-training and fine-tuning exploiting the little data available. While further pre-training gives the largest performance gains for no-resource languages, applying it directly to instruction-tuned models harms their ability to follow instructions. To address this, we start from a base model, further pre-training it on the target language, and then inject instruction-following capabilities via weight diff transfer from an instruction model. Such an approach significantly improves code generation capabilities in no-resource settings, allowing companies to cheaply deploy a specialized instruct model without dealing with the computational cost of instruction fine-tuning.