使用图灵奖励学习用户模拟器
Learning User Simulators with Turing Rewards
摘要
在交互式环境中模拟人类用户的行为，有助于推动智能助手的训练、个性化系统的评估以及社会科学领域的研究等。目前的方法通常是通过训练大型语言模型，使其能够产生与真实用户回复一致的响应，要么通过最大化对数概率，要么通过使用相似性奖励来实现这一点。我们则提出了{Turing-RL}：一种基于图灵测试的强化学习方法，用于训练用户模拟器模型。{Turing-RL}利用具有判别性的图灵奖励机制，由大型语言模型作为评判者来评估生成的响应与真实用户的回复在多大程度上可以区分开来；同时，用户模拟器模型也会学会生成与用户可能说的内容几乎无法区分的回复。在对话和Reddit论坛讨论这两个不同领域中，我们发现{Turing-RL}在大型语言模型和人类评估指标上都优于传统方法。我们的研究表明，追求可区分性而非响应一致性，才是有效训练用户模拟器的关键。
English Abstract
Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose {Turing-RL}: a Turing-Test-based reinforcement learning approach for training user simulator models. {Turing-RL} uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that {Turing-RL} consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.