MultiHashFormer：基于哈希的生成式语言模型
MultiHashFormer: Hash-based Generative Language Models
摘要
语言模型使用嵌入矩阵来表示各个符号，这些嵌入矩阵与词汇表的大小呈线性关系。为了限制参数数量，之前的研究通过将多个符号映射为单个向量来降低参数消耗，这种方法在仅包含编码器的模型中能够提高参数效率。不过，多对一的情况会阻碍其在因果语言模型中的应用。本文提出了MultiHashFormer这一新框架，该框架支持基于哈希的自回归机制。每个符号都被表示为一个唯一的哈希签名，即由多个独立哈希函数生成的短序列的离散哈希ID。哈希编码器将这种签名压缩成单个潜在向量，以便Transformer解码器进行处理。然后，哈希解码器生成下一个符号的哈希签名，再将其转换回文本形式。我们在1亿、10亿和30亿参数的不同规模下对我们的方法进行了测试，结果表明MultiHashFormer在多种基准测试中始终优于传统的Transformer语言模型。此外，我们还证明，我们的模型可以在不进行任何修改的情况下处理多语言词汇表的扩展问题，同时参数数量保持不变。
English Abstract
Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.