RepSelect：通过表示选择性实现稳健的LLM“去学习”功能
RepSelect: Robust LLM Unlearning via Representation Selectivity
摘要
让大型语言模型在遗忘特定知识和价值观的同时，不至于牺牲其整体能力，仍然是“反学习”过程中的核心挑战。不过，目前的方法很容易通过微调或少量样本提示来逆转这种遗忘效果，这表明模型的遗忘程度其实只是表面的。我们找到了问题的根源：现有方法关注的是那些既存在于被保留的知识库中，又可能被微调攻击者恢复的表示，因此反学习不仅会破坏模型的整体能力，而且很容易被逆转。我们提出了RepSelect算法——该算法通过每次更新前对权重梯度进行主成分压缩，从而分离出与特定知识相关的表示，同时保留模型的整体能力，同时限制微调所能恢复的信息范围。我们在两类知识领域进行了测试：生物危害相关知识和具有虐待倾向的知识。我们测试的模型包括密集式架构和混合专家架构四种类型：Llama 3、Qwen 3.5、Gemma 4 E4B以及DeepSeek V2 Lite。与五种常见的基准算法相比（GradDiff、NPO、SimNPO、RMU、UNDIAL），RepSelect在反学习后使模型的答案准确率降低了4到50倍，同时它对少量样本提示的攻击也表现出近乎完美的鲁棒性。因此，针对特定表示的优化是实现深度且稳健的语言模型遗忘机制的重要步骤。
English Abstract
Making large language models (LLMs) deeply forget specific knowledge and values without sacrificing general capabilities remains a central challenge in unlearning. However, current methods are easily reversed by fine-tuning or few-shot prompting, suggesting their forgetting is only shallow. We identify the root cause. Existing methods target representations shared with both the retain set and the subspace recovered by a fine-tuning attacker, making unlearning both disruptive to general capabilities and easy to reverse. We propose RepSelect (Representation Selectivity), isolates forget-set-specific representations by collapsing top principal components of weight gradients before each update, leaving general capabilities intact while limiting what fine-tuning can recover. We evaluate across two forget categories, biohazardous knowledge and abusive tendencies, and four model families spanning dense and Mixture-of-Experts architectures (Llama 3, Qwen 3.5, Gemma 4 E4B, DeepSeek V2 Lite). Compared to five popular baselines (GradDiff, NPO, SimNPO, RMU, UNDIAL), RepSelect achieves a 4-50x larger reduction in post-relearning answer accuracy than the strongest baseline, and is near-perfectly robust to few-shot prompting attacks. Targeting selective representations is thus an important step towards deep and robust LLM forgetting.