Morpheus：一种了解词形的神经分词器和土耳其语词嵌入器
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
摘要
土耳其语属于黏着语：词义由词素承载。不过，现代语言模型所使用的子词分词器则根据语料库统计来分割单词，从而将具有语义意义的后缀分开；而基于规则的分词器则无法将分词结果还原为原始文本。本文介绍了Morpheus——一种用于土耳其语的神经式词素边界模型。它既是一种无损失、能识别词形变化的分词器，又是一种词嵌入生成器。通过一种可微分的泊松-二项式动态规划方法，可以在训练过程中将每个字符的边界概率转化为软词素归属关系，而在推理时则得到精确的词段信息，无需对字符串进行任何标准化处理，因此decode(encode(w)) = w这一关系自然成立。由于该模型是神经式的，因此同样的分词过程也能生成结构化的词嵌入。在可逆分词器中，Morpheus拥有最低的字符比特率（1.425），约为子词家族中最佳词形匹配率的2倍（MorphScore宏观F1值为0.61，而{sim}为0.32）；同时，其使用的GPU内存仅为64K词汇量的子词分词器的19%。作为嵌入器，冻结后的Morpheus向量在词汇检索方面表现优异（根词族MAP值为0.85），在同根词验证方面也表现良好（ROC-AUC值为1.00），优于多语言检索器BGE-M3和BERTurk；在依赖上下文和词形变化的任务中，那些需要更多上下文信息的编码器仍然处于优势地位——这种权衡现象归因于Morpheus以词根为中心的架构设计。代码地址：https://github.com/lonewolf-rd/TurkishMorpheus；模型文件：https://hf-mirror.com/lonewolflab/Morpheus-TR-50K；交互式演示链接：https://hf-mirror.com/spaces/lonewolflab/morpheus-tr-demo。
English Abstract
Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents Morpheus, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so decode(encode(w)) = w holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character (1.425), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 0.61 vs.\ {sim}0.32), and uses {sim}19% less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP 0.85) and same-root verification (ROC-AUC 1.00), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://hf-mirror.com/lonewolflab/Morpheus-TR-50K; interactive demo: https://hf-mirror.com/spaces/lonewolflab/morpheus-tr-demo.