谁在“翻转”？自我反驳与跨模型反驳共同揭示了大型语言模型中决策的不稳定性
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
摘要
标准准确性基准通常用于测试大型语言模型在接近正确答案方面的表现，但这些基准并不适合用来评估当某个答案受到合理的反论挑战时，模型是否仍然能够保持正确的回答。我们提出了一种用于评估答案稳定性的方法：首先让模型正确回答一个选择题，然后用一个合理的反论来质疑模型的答案，从而判断模型是否会改变其回答。这种方法的优点在于：a) 可以将争论性内容与明显的社会压力分离开来；b) 可以针对不同模型的争论长度、模型自身的归因方式以及不同模型的来源进行差异化处理。在七个前沿模型和57个MMLU测试对象中，模型改变答案的比例介于17.5%到97.3%之间，这表明模型稳定性存在很大的差异，而这些差异仅通过准确性指标是无法完全捕捉到的。我们发现，模型自身的归因机制会显著增加模型改变答案的概率（平均增加7.1个百分点，最高可达18.7个百分点）。此外，将多个模型的错误答案整合在一起，并选择其中最有效的答案作为问题中的挑战因素，比依赖单一模型来生成挑战要更有效。我们还构建了一个名为MaxFlip的挑战集，该挑战集能够将模型改变答案的概率提高多达23.6个百分点。我们现在公开了这套评估方法、相关记录以及MaxFlip挑战集，以辅助对模型稳定性的评估，同时与标准的准确性基准一起使用。这些材料可以在以下链接找到：https://github.com/nafisenik/WhoFlips 和 https://hf.co/datasets/nafisehNik/WhoFlips。
English Abstract
Standard accuracy benchmarks are designed to test how closely large language models (LLMs) approach correct answers, but are not suitable for testing whether LLMs stick with a correct answer when that answer is challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean +7.1pp, up to +18.7pp). Also, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. We further construct MaxFlip, a curated challenge set that amplifies flips by up to +23.6pp over standard self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at https://github.com/nafisenik/WhoFlips and https://hf.co/datasets/nafisehNik/WhoFlips.